The data moat
The longest-running continuous ecommerce intelligence dataset in Southeast Asia. Built on infrastructure that cracks Shopee's anti-bot defenses — and has been doing it since 2020.
Why the depth matters
Six years tells you how categories move across peak seasons, how price wars resolve, how new entrants grow or fail, and which market share gains are structural versus promotional.
Magpie's taxonomy has been maintained continuously — a query about Baby Care in 2024 maps correctly to Baby Care in 2020, even through two Shopee category restructures.
Bar = relative historical depth. Shopee Indonesia has the deepest archive.
The technical moat
Anti-bot defenses
Dynamic rate limiting, IP rotation detection, behavioural fingerprinting, and frequent API schema changes. Most competitors break within days. We've maintained continuous operation for six years.
Unbroken since 2020
No gaps. No resets. The taxonomy has been maintained through two Shopee category restructures — enabling true six-year trend analysis without manual reconciliation.
Multi-language signals
Review sentiment in Bahasa Indonesia, Filipino, Thai, and Vietnamese. How Owl detects 'palsu', 'kw', and 'peke' in counterfeit listings — real local signals, not proxies.
Stable taxonomy layer
10 million product SKUs mapped to a human-maintained taxonomy. The same FMCG category in 2020 is the same category today — enabling cross-year analysis without data science overhead.
How it works
Four stages. Fully automated. Running continuously since 2020.
Automated scraping across five platforms. Anti-bot handling, rate limits, proxy rotation. Monthly cycles across all markets.
Prices normalised across promotional mechanics. Sold counts reconciled against baselines. Duplicates removed. Flash sale flags attached.
AI-assisted taxonomy with human review. 10 million product SKUs mapped to stable categories. Cross-platform SKU matching applied.
Platform coverage
Dates show the first month of available data (YYYY-MM). — means that category is not yet available for that marketplace and country, or the platform does not operate there. All platforms refresh monthly, with higher-frequency capture for actively monitored categories. Fields captured per SKU include price, stock, units sold, reviews, seller type, category, rank, and promotions (varies by platform).
Live data
Free monthly snapshots from the universe — market share, GMV trends and category movers. Exact figures and full history are in Farsight.
FAQ
Magpie IQ has operated a continuous Shopee scraping pipeline since 2020 — longer than any comparable SEA-native intelligence provider. The dataset now contains 78 billion data points across five platforms, with Indonesia as the primary and deepest market.
Through six years of continuous engineering investment — proxy management, behavioural mimicry, rate limit negotiation, and rapid response to platform changes. This is maintained as an ongoing engineering function, not a one-time build.
Shopee's sold count updates when buyers confirm receipt — creating a lag between transaction and recorded sold. Magpie reconciles these against historical baselines and flags anomalies. Data notes surface any caveats in Farsight answers.
Three access points: Farsight (natural-language AI interface), Nest API (direct REST access), or managed Looker Studio dashboards. Email sales@magpieiq.com to discuss.