--- name: magpie-ecommerce-data description: Query Southeast Asian ecommerce data through Magpie's two APIs. Use for questions about brands, market share, GMV, pricing, assortment or sellers on Shopee, Tokopedia, TikTok Shop, Lazada and Blibli — whether the question is historical ("who gained share last quarter", "what price bands exist") or live ("what does this product cost right now", "who is selling my brand"). Covers the Data API (a maintained monthly panel, delivered as bulk exports) and the Scraping API (on-demand collection from live marketplace pages). --- # Magpie ecommerce data Two APIs, one key. **The Data API** serves a maintained monthly panel — already scraped, cleaned, brand-labelled and deduplicated. **The Scraping API** fetches live marketplace pages on request. Ask which time horizon the question has. "How did share move" is the panel. "What does it cost today" is scraping. Some questions need both: use the panel to pick the SKUs worth watching, then scrape those. --- ## Setup ```bash export MAGPIE_API_KEY="mgk_live_..." # your key; never commit it, never echo it DATA=https://api.magpieiq.com/data SCRAPE=https://api.magpieiq.com/scraping AUTH="Authorization: Bearer $MAGPIE_API_KEY" ``` One key works on both hosts. Get one at . **`data.magpieiq.com` is the documentation and billing portal, not an API host.** Calls to it return 404. The API hosts are the two above. Live specs, fetch these if anything here looks stale: · --- ## House rules Every one of these exists because breaking it produced a wrong answer or an unnecessary charge. **1 · Never spend before costing.** Every Data API export accepts `"estimate": true`, which returns exact rows, credits and dollars and charges nothing. Always estimate, show the figure to the user, and wait for approval before submitting for real. The gap between a full category export and a `top` slice routinely exceeds a factor of a hundred. **2 · Ask the catalogue what exists.** `GET /v1/exports/catalog?country=ID` returns the exact country codes, platform names and `category_3` values that can be exported, with a month range for each. Use those strings verbatim. An invented category name errors; omitting `category_3` entirely exports every category in that market, which is expensive and rarely intended. The call costs 1 credit. **3 · Know what runs unattended.** `get_pc`, `get_pc_bulk` and all Data API exports have no approval step and no working-hours window, so an automated schedule can rely on them. `search-items`, `merchant-items` and `variant_sold_v1` are approval-gated and process only 08:00–18:00 GMT+8. Never build an unattended job on those three. **4 · Poll with backoff, and hand back gated jobs.** 10s → 30s → 60s. An approval-gated job returns `awaiting_approval`; report the `job_id` to the user and stop, rather than polling for hours. Approval is manual and has a 24h timeout. **5 · Persist raw results before analysing.** Write the downloaded payload to disk first. Re-running a job costs credits again. **6 · Report what failed.** Everything is asynchronous. Multi-URL jobs sometimes fail the last few URLs in submission order, so put the URLs that matter most first, and never present a total without saying how many frames are missing from it. **7 · Insufficient balance returns `402`.** Balance is checked at submit, not at completion. --- ## What a call costs | Endpoint family | Approval | Hours | Cost | |---|---|---|---| | Data API `/v1/exports` | none | any | 15 credits/row · 50 for `top` rows | | Data API `/v1/exports/catalog` | none | any | 1 credit | | Data API estimate (`"estimate": true`) | none | any | **free** | | `shopee/get_pc`, `get_pc_bulk` | none | any | 8 credits/product | | `shopee/search-items` | **required** | 08:00–18:00 GMT+8 | 6 credits/URL | | `shopee/merchant-items` | **required** | 08:00–18:00 GMT+8 | 6 credits/URL | | `shopee/variant_sold_v1` | **required** | 08:00–18:00 GMT+8 | 10 credits/item | | `tokopedia/*`, `blibli/*` | none | any | 1 credit/request | | `lazada/*` | none | any | 6 credits/request item | Credits price at USD 0.00125 each at the entry tier; larger deposits lower it. Authoritative rate card: `variant_sold_v1` deducts **only on approval** — rejection or a 24h timeout costs nothing. --- ## Data API Markets: `ID`, `TH`, `VN`, `SG`, `PH`, `MY`. Coverage as at 2026-09-08: Indonesia runs from 2020-11 and carries 121 categories; the other five markets run from 2022-11 and carry twelve to fifteen. Every market's latest month is 2026-06. Always confirm against the catalogue rather than trusting these figures — they move monthly. ### The four calls ```bash # 1. What exists (1 credit) curl "$DATA/v1/exports/catalog?country=ID" -H "$AUTH" # 2. What it would cost (free, charges nothing) curl -X POST "$DATA/v1/exports" -H "$AUTH" -H "Content-Type: application/json" \ -d '{"country":"ID","category_3":["Shampoo"],"platforms":["Shopee"], "date_from":"2026-01","date_to":"2026-06","top":100, "format":"csv","estimate":true}' # 3. The same body without estimate → returns an export_id # 4. Poll until the files are ready curl "$DATA/v1/exports/{export_id}" -H "$AUTH" ``` ### Request fields | Field | Notes | |---|---| | `country` | One of the six market codes | | `category_3` | Array. Use catalogue strings verbatim. Omit = every category, expensive | | `platforms` | Array of exact platform values. Omit = all | | `date_from`, `date_to` | `YYYY-MM`, both required | | `top` | Top-N highest-GMV rows per atom. **Caps at 1000.** Omit = full category | | `format` | `parquet` (default) or `csv` | | `estimate` | `true` = free dry run | | `idempotency_key` | Optional; stops a retry double-charging | ### What an estimate returns Real response, `Shampoo` / Shopee / ID / 2026-01 to 2026-06: ```json {"estimate": {"rows": 374169, "credits": 5612535, "usd": 7015.67, "parts": 6, "price_basis": {"credits_per_row": 15.0, "usd_per_credit": 0.00125, "credits_per_row_top": 50.0}, "top": {"n": 100, "rows": 600, "credits": 30000, "usd": 37.5}}, "charged": 0} ``` Note the `top` block: the same scope costs **USD 7,015 in full and USD 37.50 at `"top": 100`**. Show the user both before spending. ### Two catalogue behaviours that will trip you **It returns six platform values, not five.** Indonesia lists `Tokopedia | Shop` alongside `Tokopedia`. That is a surface inside Tokopedia, not a sixth marketplace. Use the string verbatim when querying; do not report it as another marketplace in a summary. **One category has no name.** The Indonesian catalogue contains an entry whose `category_1`, `category_2` and `category_3` are all empty strings. Skip it when looping rather than trying to export it. --- ## Scraping API Regions: `id`, `sg`, `th`, `ph`, `my`, `vn`, `tw`, `br`. All jobs are submit-then-poll. Lazada is the exception: `POST /v1/lazada/` returns `{status, cost, upstream}` more or less directly, with `GET /v1/lazada/retrieve/{task_id}` for anything deferred. | You want | Endpoint | |---|---| | One Shopee product page | `POST /v1/shopee/get_pc/submit` — `{region, item_id, shop_id}` | | Many Shopee product pages | `POST /v1/shopee/get_pc_bulk/submit` — multipart CSV | | Shopee search / category listing | `POST /v1/shopee/search-items/submit` — `{urls[], max_pages}`, ≤50 URLs | | Everything in a Shopee shop | `POST /v1/shopee/merchant-items/submit` — same shape | | Shopee per-variant sold counts | `POST /v1/shopee/variant_sold_v1/submit` — `{items[{item_id, shop_id, country}]}`, ≤50,000 | | Tokopedia PDP / search / shop | `POST /v1/tokopedia/{pdp,search,merchant}/bulk/submit` — ≤50 items/job | | Blibli PDP / shop | `POST /v1/blibli/{pdp,merchant}/submit` | | Lazada products, reviews, sellers, categories | `POST /v1/lazada/` | Retrieve with `GET /v1///retrieve/{job_id}` (or `{task_id}` for `get_pc`). A Shopee product URL ends in `-i.{shop_id}.{item_id}` — that is where those two values come from. Sorting and filtering are expressed **in the URL**, exactly as on the site: `?page=0&sortBy=sales` for best-sellers, `sortBy=ctime` for newest. The market is set by the domain: shopee.co.id, shopee.com.my, shopee.co.th, shopee.ph, shopee.sg, shopee.vn. ### Parsing Shopee responses These field paths are **not in the OpenAPI spec** — `result` is typed as an open object. They come from running the endpoints. Verify against an actual response before relying on one, and say so if a path is missing rather than silently reporting zero. - Iterate `response['search_items']['items']` directly. Do not recurse looking for fields. - Product name is on the wrapper at `item_card_displayed_asset.name`, not the inner object. - Units sold is the numeric `item_card_display_sold_count.monthly_sold_count`. The `_text` version is a bucket like "10k+" and makes every listing look identical. - Price is `item_card_display_price.price`, **divided by 100000**. - Shop is `item_data.shop_data.shop_name`, id `item_data.shopid`, location `item_card_displayed_asset.shop_location`. - **Deduplicate by `itemid` before summing anything.** `merchant-items` behaves differently: its numeric sold field is `0`, so read `monthly_sold_count_text`, which is exact there. It also repeats the same item across a shop's category tabs, so deduplicate or you will overcount by roughly half. Lazada endpoints each validate their own undocumented URL pattern, and a URL accepted by one is often rejected by another. Read the error and try the pattern it names rather than assuming the endpoint is broken. --- ## Recipes ### 1.1 · Category market share and trend *Who leads a category, how concentrated it is, how that has moved.* — Data API Call the catalogue and confirm the category and market are available and for which months. Estimate, show rows/credits/dollars, wait for approval, then submit and poll. From the data report: the ten largest brands by GMV and their share in the most recent month; how each share moved since the first month; realised price per unit for each against the category average as an index with the average at 100; and any brand whose share moved more than a third either way. Flag any month whose row count is sharply out of line with its neighbours — that is a collection artefact, not a market move, and should be said rather than reported as a trend. ### 1.2 · Price architecture for a new entry *What price bands exist, and which are crowded.* — Data API Export the most recent full month at `"top": 500`. Compute realised price per unit per brand as GMV divided by units. Bucket brands into price deciles against the category average and report, per decile, how many brands sit there and what share of category GMV they hold together. Look for a band holding meaningful demand but few competing brands. **Say explicitly if there is no such gap rather than manufacturing one.** ### 1.3 · Is this category worth entering *Size, concentration, and whether the top is defended.* — Data API Export the most recent month at `"top": 100`. Report the combined share of the top ten brands, the number of brands needed to reach half of category GMV, and the realised price index of the top three against the category average. Then classify: concentrated and premium (leaders hold high share at high price), concentrated and cheap (leaders hold share by undercutting), or fragmented (no brand holds meaningful share). Each implies a different entry cost — give the reasoning, not just the label. ### 2.1 · Price match: is anyone undercutting us *Whether a published price is genuinely the lowest.* — Scraping, **runs unattended** For each product the user supplies with an offer price and a Shopee URL, extract `shop_id` and `item_id` from the URL, submit to `get_pc`, poll, and read the current price divided by 100000. Return a table of every product where the Shopee price is at or below theirs, with both prices and the gap. The payload also carries `is_lowest_price_at_shopee` — report it alongside each row, since it answers the question directly. Use `get_pc` only. At scale, use `get_pc_bulk` with a CSV, and `get_pc_bulk/schedules` with a cron to run it nightly. 8 credits per product. ### 2.2 · Who is selling in this category right now *The shops that dominate a category today, official versus reseller.* — Scraping, **approval-gated** Submit `search-items` with Shopee search URLs sorted by sales, several keywords covering the category, ≤50 URLs. Poll, download each frame, parse per the rules above. Compute GMV per listing as units × price, aggregate by shop, and report the twenty largest shops with their share of the total observed, listing count, average realised price and location. Separate official brand stores from resellers where the shop name makes that clear. Report how many submitted URLs returned data and how many failed. ### 2.3 · Unauthorised sellers and channel control *Who is selling your brand without permission, and at what price.* — Scraping, **approval-gated** Run 2.2 using the brand name and its common misspellings as keywords. Aggregate by shop. Given a list of authorised distributors, produce two tables: every shop selling the brand that is not on the list, with observed GMV, listing count and location, sorted by size; and any authorised shop whose realised price sits more than twenty per cent below the stated floor. **Do not accuse anyone of counterfeiting.** Report what the listing data shows — who is selling, at what price, from where. ### 2.4 · Enumerate a competitor's whole shop *A rival's full catalogue, pricing, and what actually sells.* — Scraping, **approval-gated** Submit `merchant-items` with shop URLs, ≤50 per request, and poll. Remember the numeric sold field is 0 here — read `monthly_sold_count_text`. Deduplicate by `itemid` first or the total roughly doubles. Report listing count, total observed monthly GMV, the twenty listings carrying most of it, and the price distribution. **Do not compare this shop's total against a market total built from keyword search.** The shop is completely enumerated and the market is not, so a share computed that way is a ceiling, not an estimate. If asked for a share, say so. ### 2.5 · New product and assortment watch *What competitors have just launched.* — Scraping, **approval-gated** Run 2.2 with `sortBy=ctime` instead of `sortBy=sales`. Run weekly and keep the previous week's `itemid` list. Each week report listings new since the last run, grouped by shop, with price and any units already recorded. Highlight any new listing that took meaningful volume immediately — that usually means a launch with paid support behind it. ### 2.6 · SKU-level variant demand *Which size, shade or pack actually sells.* — Scraping, **approval-gated** Submit `variant_sold_v1` with `{items: [{item_id, shop_id, country}]}`. One country per request, up to 50,000 items, billed per item. Report sold count per variant, ranked, with each variant's share of the product total, and say whether demand concentrates in one or two variants or spreads evenly — that determines whether a range needs pruning. ### 2.7 · Review harvesting for product feedback *What buyers complain about, in their own words.* — Scraping Submit `POST /v1/lazada/product-reviews-v2` with `{"requests":[{"url":"..."}], "language":"en"}`, then poll `GET /v1/lazada/retrieve/{task_id}`. If the URL is rejected, read the error — each Lazada endpoint validates its own pattern. Group reviews by star rating, summarise recurring complaints in the one and two star reviews and recurring praise in the five star ones. Quote a few representative lines verbatim rather than paraphrasing everything, and say how many reviews were read. --- ## Choosing between them | The question | Use | Recipe | |---|---|---| | Brand shares and how they moved over months | Data API | 1.1 | | Price bands, where a category is crowded | Data API | 1.2, 1.3 | | Current price on specific listings | Scraping · `get_pc` | 2.1 | | Who is selling right now | Scraping · `search-items` | 2.2, 2.3 | | A competitor's full catalogue | Scraping · `merchant-items` | 2.4 | | A market or category the panel does not cover | Scraping | 2.2 | | Anything that must run unattended on a schedule | Scraping · `get_pc_bulk`, or Data API | 2.1 | **The panel is not universal.** Before promising a market and category combination, check the catalogue. Coverage is category-by-market, not country-wide, and a category present in one market is frequently absent in the next. Where the panel holds nothing, the Scraping API still reaches the marketplace directly. --- Docs · rate card · machine-readable brief Use subject to the Magpie IQ terms of service.