1. Collection
Marketplace data is gathered continuously.
The pipeline collects product, seller, pricing, review, and category information across supported marketplaces and markets.
Methodology
This page explains how Magpie IQ collects, cleans, categorizes, normalizes, and interprets ecommerce data across Southeast Asia so teams can trust the outputs behind market share, pricing, and assortment analysis.
Methodology steps
1. Collection
The pipeline collects product, seller, pricing, review, and category information across supported marketplaces and markets.
2. Normalization
SKUs, brands, sellers, and category structures are reconciled so teams can compare like with like.
3. Interpretation
Outputs are shaped into market share, pricing, assortment, and seller-intelligence views that commercial teams can act on.
What this protects against
Why it matters
When product names, seller structures, prices, and category trees are cleaned consistently, the output becomes decision-grade. That discipline is part of what makes Magpie IQ more credible for real commercial work.
Data collection
Pipeline infrastructure
Magpie IQ runs a continuous automated scraping pipeline across Southeast Asia's major ecommerce platforms. The system has been operational since 2020 on Shopee and Tokopedia, with coverage expanding to Lazada and Blibli in 2021, regional Shopee and Lazada markets from 2021, and TikTok Shop in 2025. Each platform presents distinct collection challenges. Shopee in particular operates some of the most aggressive anti-scraping infrastructure among Southeast Asia marketplaces — a technical moat that means less sophisticated data providers cannot maintain consistent collection. Magpie IQ's pipeline was built specifically for this environment, and six years of continuous operation have produced the operational depth required to stay current.
Collection frequency
Data is refreshed at a monthly minimum across all platforms, with higher-frequency capture available for categories under active monitoring — for example, during major promotional periods (Harbolnas, 9.9, 12.12) or when a client has flagged a competitor category for close tracking. Monthly cadence is sufficient for structural market share analysis. Campaign-period dynamics require tighter snapshot intervals to separate promotional peaks from baseline performance. The pipeline architecture supports both modes.
SKU-level granularity
Collection operates at the individual SKU level — not aggregated category feeds or seller-level summaries. Each data point captures product identity, pricing, seller attribution, review count, category placement, and sales volume signal. The result is 78 billion data points accumulated since 2020, covering 10 million+ active and historical SKUs across five platforms. This granularity is what enables SKU-level competitive analysis, exact pricing surveillance, and sub-category share calculations that are not dependent on platform-reported aggregate numbers.
GMV calculation
Sales Value (GMV) = Final post-discount price × Units sold (Terjual)
Price is held constant at the latest snapshot captured within the reporting period. This means that if a SKU was priced at Rp 85,000 at the time of capture and the platform's Terjual counter shows 1,200 units sold, GMV for that SKU is recorded as Rp 102,000,000 for the period. Units sold is sourced from the platform's cumulative "terjual" counter — the publicly visible sold count. Magpie IQ calculates the incremental change in this counter between snapshots to derive period-specific unit volume.
This approach differs from platform-reported GMV. Platform-reported figures often include cancelled orders, returns, and pre-shipment adjustments that are not publicly visible at the product listing level. Magpie IQ's GMV is a floor estimate based on observable data — it is consistent and comparable across time and across platforms, but it is not a certified sales figure and should not be treated as one.
TikTok Shop Indonesia data in Magpie IQ is available from 2025, with categories added through the year as each reached coverage and data-quality thresholds (Pet from February 2025, Beauty and Mom & Baby from March 2025, Groceries from July 2025, Health from October 2025). The TikTok Shop dataset is structured for consistent period-over-period comparison from each category's start month. Cross-platform analyses that include TikTok Shop should use a consistent platform set and note that TikTok Shop history is shorter than Shopee, Tokopedia, Lazada, and Blibli.
When adding a new platform to a cross-platform analysis, historical comparability changes: a 2021 total across Shopee and Lazada is not directly comparable to a 2025 total across all five platforms. Period comparisons should always use a consistent platform set. Magpie IQ reports always specify the platform scope used for any share or GMV calculation.
Classification and normalisation
Magpie IQ tracks 18,000+ brands across five platforms. The challenge is that the same brand may appear under dozens of listing formats: different seller accounts, varied product title strings, mixed official and reseller channels, and inconsistent category placements by the platforms themselves. A major FMCG brand on Shopee Indonesia can have 15 or more distinct listing formats across official stores, third-party resellers, and platform-operated grocery channels. Adding these up without normalisation produces an artificially low brand share — because many listing variants are missed — or an inflated one, if duplicates are counted twice.
Magpie IQ's normalisation layer resolves this. Products are matched to a canonical SKU record. Brand attribution is applied consistently across listing variants. The result is a single, aggregated brand GMV figure that is the actual competitive share — not a partial count of easily identifiable official store listings.
Every seller in the dataset is classified into one of three types: OS (Official Store — brand-operated), C2C/Reseller (third-party seller distributing the brand's products), and Mart (platform-operated grocery or convenience channel, such as Shopee Mart or Tokopedia NOW). This classification matters for commercial analysis: a brand's "true" ecommerce performance is its Official Store volume. Including all reseller channels inflates the number and mixes in distributor sell-through, which reflects different margin structures and demand signals.
For category share calculations, Magpie IQ can produce both an all-seller view (total category GMV regardless of seller type) and an official-store-only view. Many brand strategy questions — for example, how much of a competitor's growth is coming from their own channel versus resellers — require access to both cuts of the data.
Limitations
Magpie IQ publishes its limitations explicitly because serious teams need to know what the data does and does not measure. Using ecommerce data without understanding its boundaries leads to incorrect conclusions.
What Magpie IQ does not measure
Offline sales, wholesale distributor volumes, and modern trade (supermarket, minimarket) performance are not included. Magpie IQ is an ecommerce marketplace intelligence system — it does not integrate point-of-sale, Nielsen retail audit, or distributor sell-in data. In-app advertising spend on the platforms is not tracked. Order cancellations and customer returns, which are not visible in the Terjual counter after the fact, are not accounted for in GMV calculations.
Units sold interpretation
The "units sold" figure derived from the platform Terjual counter reflects transactions completed on the platform. In categories with significant B2B or bulk purchasing — for example, baby formula sold in multi-pack configurations to resellers, or pet food sold in 20kg catering packs — unit counts will include non-consumer purchases. This inflates unit volume relative to household consumption in those sub-categories. Category teams should apply product-level judgment when interpreting unit data in bulk or institutional SKUs.
Platform expansion effects
When a new platform is added to Magpie IQ's coverage, total market GMV figures change. A 2021 market size figure based on Shopee and Lazada is not directly comparable to a 2025 figure based on five platforms. Magpie IQ reports always specify the platform scope. When clients use the data for trend analysis over multi-year periods, Magpie IQ recommends holding the platform set constant or clearly annotating the scope change in any presentation of market size.
Platform coverage
The table below shows data availability by category, marketplace, and country — the month from which each category's data is available in Magpie IQ (YYYY-MM).
| Platform | Country | Beauty | Electronics | F&B | Groceries | Health | Household | Mom & Baby | Pet | Tobacco |
|---|---|---|---|---|---|---|---|---|---|---|
| Shopee | Indonesia | 2021-01 | 2020-10 | 2021-06 | 2021-06 | 2020-10 | 2021-05 | 2020-03 | 2022-01 | 2023-01 |
| Malaysia | — | 2021-08 | 2025-09 | 2025-09 | — | — | 2025-09 | 2025-09 | — | |
| Philippines | — | 2021-08 | 2025-09 | 2025-09 | — | — | 2025-09 | 2025-09 | — | |
| Singapore | 2025-06 | 2021-08 | 2025-09 | 2025-09 | 2025-06 | 2025-06 | 2025-06 | 2025-06 | — | |
| Thailand | 2025-06 | 2021-08 | 2025-09 | 2025-09 | 2025-06 | 2025-06 | 2025-06 | 2025-06 | — | |
| Vietnam | — | 2021-08 | 2025-09 | 2025-09 | — | — | — | 2025-09 | — | |
| Lazada | Indonesia | 2023-03 | 2023-04 | 2023-04 | 2023-04 | 2023-04 | 2022-04 | 2023-01 | 2022-02 | 2023-01 |
| MY, PH, SG, TH, VN | — | 2021-08 | — | — | — | — | — | — | — | |
| Tokopedia | Indonesia | 2021-02 | 2020-04 | 2021-06 | 2021-06 | 2020-10 | 2021-05 | 2020-05 | 2022-01 | 2023-01 |
| Blibli | Indonesia | 2022-04 | 2021-03 | 2022-01 | 2022-01 | 2022-02 | 2022-01 | 2021-03 | 2022-02 | 2023-01 |
| TikTok Shop | Indonesia | 2025-03 | — | — | 2025-07 | 2025-10 | — | 2025-03 | 2025-02 | — |
Dates show the first month of available data (YYYY-MM). — indicates the category is not yet available for that marketplace and country, or the platform does not operate in that market. Coverage is added as platform scale and data quality thresholds are met. Contact Magpie IQ for current coverage status in specific markets.
FAQ
No. Raw data is one layer. Magpie IQ also provides cleaned, normalized views for market share, pricing, assortment, and seller analysis.
Because serious teams should expect to see how the work is done. Magpie IQ is strongest when the data, process, and limitations are explicit.