What ecommerce data does the Magpie IQ Nest API provide?

The Nest API provides SKU-level ecommerce data across Shopee, Lazada, TikTok Shop, Tokopedia, and Blibli. Fields include: SKU name, brand, platform, seller type (Official Store or C2C), GMV, units sold, average price, listed price, discount percentage, rating, review count, category, sub-category, and seller city. Historical data is available from 2020 for Shopee and Tokopedia, from 2021 for Lazada and Blibli, and from 2025 for TikTok Shop.

Which countries does the Magpie IQ API cover?

The Nest API covers Indonesia (primary market with deepest coverage), Malaysia, Philippines, Thailand, Vietnam, and Singapore across Shopee and Lazada. TikTok Shop coverage spans Indonesia, Thailand, Vietnam, and Philippines. Tokopedia and Blibli cover Indonesia only.

How far back does Magpie IQ ecommerce historical data go?

Magpie IQ's Shopee Indonesia data is available from 2020, making it the longest-running continuous ecommerce dataset in Southeast Asia. Tokopedia data is also available from 2020, Blibli from 2021, Lazada Indonesia from 2022, and TikTok Shop from 2025. Shopee and Lazada coverage outside Indonesia (MY, PH, TH, VN, SG) begins in 2021.

Nest API · Reference

API reference

Complete endpoint reference for the Nest API (Magpie Scraping Service API v1.0.0). Submit scraping jobs, poll status, and download results across Shopee variant-sold, PDP, search, and merchant endpoints.

Open interactive docs ↗ ← Back to Nest API

Getting started

Base URL & authentication

All requests go to the base URL below over HTTPS. Authenticate with a Bearer token (your API key) in the Authorization header. Request access from the Nest API page.

Base URL	https://nest-api.magpieiq.com
Auth header	Authorization: Bearer <your-api-key>

Endpoints

All endpoints

POST/shopee/variant_sold_v1/submit

Submit a new scraping job

Accepts up to **50,000 items**, one country per request, and chunks each group into batches of 50 (for background processing). This endpoint only gets processed during working hours (8am-6pm GMT+8) **Cost: 10 credits per item_id.** Balance is checked upfront — returns `402` if insufficient.

Starts in `awaiting_approval` with **no credits deducted yet**. The team is notified and waits for approval (24h timeout). **Credits are only deducted when approved.** On rejection or timeout, the job ends in `rejected` with no charge.

Returns `202 Accepted` with a `job_id` for polling via `GET /retrieve/{job_id}`.

Request body · SubmitRequest

Field	Type	Description
items *	array<SubmitItem>	Array of items to scrape. Between 1 and 50,000 entries.

Responses

202 — Job accepted; awaiting approval or already queued. · SubmitResponse
401 — Missing or invalid Bearer token. · ErrorResponse
402 — Insufficient quota balance. Response includes `required`, `current`, and `shortfall`. · QuotaErrorResponse
422 — Payload validation failed (bad country code or schema).

GET/shopee/variant_sold_v1/retrieve/{job_id}

Retrieve job status and download URLs

Returns the consolidated status of a submitted job, along with per-batch details and **time-limited signed URLs** for any `SUCCESS` batches. This endpoint only gets processed during working hours (8am-6pm GMT+8)

Parameters

Name	In	Type	Description
job_id *	path	string (uuid)

Responses

200 — Job found; returns status and per-batch details. · JobRetrieveResponse
401 — Missing or invalid Bearer token. · ErrorResponse
404 — Job not found, or not owned by the authenticated user. · ErrorResponse
422 — Validation Error · HTTPValidationError

GET/shopee/variant_sold_v1/download/{job_id}

Download the merged result for a job

Returns counters and a **signed URL** to the merged `full_{job_id}.json` containing every item in the job. This endpoint only gets processed during working hours (8am-6pm GMT+8)

If the job has not yet been completed, `download_url` will be `null` and the counts will reflect current progress.

Parameters

Name	In	Type	Description
job_id *	path	string (uuid)

Responses

200 — Job found; signed URL included when available. · JobDownloadResponse
401 — Missing or invalid Bearer token. · ErrorResponse
404 — Job not found or not owned by caller. · ErrorResponse
422 — Validation Error · HTTPValidationError

GET/shopee/variant_sold_v1/download/{job_id}/item/{item_id}

Download a single item result

Returns the scraped payload for a single item directly from the database. This endpoint only gets processed during working hours (8am-6pm GMT+8)

`data` is populated only when `status == SUCCESS`. For `FAILED` items, `error_log` explains the reason.

Parameters

Name	In	Type	Description
job_id *	path	string (uuid)
item_id *	path	string

Responses

200 — Item found. · ItemDownloadResponse
401 — Missing or invalid Bearer token. · ErrorResponse
404 — Job or item not found, or not owned by caller. · ErrorResponse
422 — Validation Error · HTTPValidationError

POST/shopee/get_pc/submit

Submit a PDP tier 2 task

Creates one asynchronous Shopee PDP tier 2 task (get_pc). Cost: 8 credits per request.

Request body · ShopeeGetPcSubmitRequest

Field	Type	Description
region *	string	Shopee region code.
item_id *	string	Shopee item ID.
shop_id *	string	Shopee shop ID.

Responses

202 — Task accepted. · ShopeeGetPcSubmitResponse
401 — Missing or invalid Bearer token. · ErrorResponse
402 — Insufficient quota balance. · QuotaErrorResponse
422 — Payload validation failed.
502 — Failed to create task. · ErrorResponse

GET/shopee/get_pc/retrieve/{task_id}

Retrieve a PDP tier 2 task

Polls one owned PDP tier 2 Hoarder task and returns the GCS JSON result when ready.

Parameters

Name	In	Type	Description
task_id *	path	integer

Responses

200 — Task status or result. · ShopeeGetPcRetrieveResponse
401 — Missing or invalid Bearer token. · ErrorResponse
404 — Task not found. · ErrorResponse
502 — Failed to retrieve task or result. · ErrorResponse
422 — Validation Error · HTTPValidationError

POST/shopee/search-items/submit

Submit Shopee search/category scraping tasks

Submits multiple URLs to scrape items from Shopee search or category pages. This endpoint only gets processed during working hours (8am-6pm GMT+8) **Cost: 6 credits per URL.** Balance is checked upfront. Up to 50 urls per request.

The job goes through the approval by our team before processing. Poll `GET /retrieve/{job_id}` to check progress.

Request body · SearchItemsSubmitRequest

Field	Type	Description
urls *	array<string>	List of Shopee search or category URLs to scrape.

Responses

202 — Job accepted; awaiting approval. · PgntSubmitResponse
401 — Missing or invalid Bearer token. · ErrorResponse
402 — Insufficient quota balance. · QuotaErrorResponse
422 — Payload validation failed.

GET/shopee/search-items/retrieve/{job_id}

Retrieve search/category job status

Returns the current status and per-URL results when complete. This endpoint only gets processed during working hours (8am-6pm GMT+8)

Parameters

Name	In	Type	Description
job_id *	path	string (uuid)

Responses

200 — Job status. · PgntRetrieveResponse
401 — Missing or invalid Bearer token. · ErrorResponse
404 — Job not found. · ErrorResponse
422 — Validation Error · HTTPValidationError

POST/shopee/merchant-items/submit

Submit Shopee merchant/shop scraping tasks

Submits multiple URLs to scrape all items from Shopee merchant/shop pages. This endpoint only gets processed during working hours (8am-6pm GMT+8) **Cost: 6 credits per URL.** Balance is checked upfront. Up to 50 urls per request.

The job goes through the approval by our team before processing. Poll `GET /retrieve/{job_id}` to check progress.

Request body · MerchantItemsSubmitRequest

Field	Type	Description
urls *	array<string>	List of Shopee merchant/shop page URLs to scrape.

Responses

202 — Job accepted; awaiting approval. · PgntSubmitResponse
401 — Missing or invalid Bearer token. · ErrorResponse
402 — Insufficient quota balance. · QuotaErrorResponse
422 — Payload validation failed.

GET/shopee/merchant-items/retrieve/{job_id}

Retrieve merchant job status

Returns the current status and per-URL results when complete. This endpoint only gets processed during working hours (8am-6pm GMT+8)

Parameters

Name	In	Type	Description
job_id *	path	string (uuid)

Responses

200 — Job status. · PgntRetrieveResponse
401 — Missing or invalid Bearer token. · ErrorResponse
404 — Job not found. · ErrorResponse
422 — Validation Error · HTTPValidationError

GET/shopee/pgnt/jobs/{job_id}/items/{item_token}/data

Download PGNT result JSON for a specific URL

Returns the JSON payload collected for the given URL once it is marked SUCCESS. This endpoint only gets processed during working hours (8am-6pm GMT+8)

Parameters

Name	In	Type	Description
job_id *	path	string (uuid)
item_token *	path	string

Responses

200 — Result JSON.
400 — Invalid item token. · ErrorResponse
401 — Missing or invalid Bearer token. · ErrorResponse
404 — Result not available. · ErrorResponse
502 — Failed to fetch result data. · ErrorResponse
422 — Validation Error · HTTPValidationError

GET/health

Health check

Simple liveness probe. Returns `{"status": "ok"}` if the service is running.

Responses

200 — Successful Response

Reference

Data models

BatchDetail

Field	Type	Description
batch_id *	string (uuid)	Unique identifier for this batch (child of the parent job).
country *	string	Country code this batch was dispatched to.
status *	string	Batch status: `PENDING`, `SUCCESS`, or `FAILED`.
download_url	string	Time-limited GCS signed URL to the result JSON. Only present when `status == SUCCESS`. Regenerated on every `/retrieve` call and expires in 7 days.

ErrorResponse

Field	Type	Description
detail *	string	Human-readable error message.

HTTPValidationError

Field	Type	Description
detail	array<ValidationError>

ItemDownloadResponse

Field	Type	Description
job_id *	string (uuid)
item_id *	string	The item_id the client originally submitted.
shop_id *	string
country *	string
status *	string	`PENDING`, `SUCCESS`, or `FAILED`.
data	object	The scraped payload for this item, as produced by the worker. `null` unless status is `SUCCESS`.
error_log	string	Error details if `status == FAILED`.

JobDownloadResponse

Field	Type	Description
job_id *	string (uuid)	The job identifier.
status *	string	Job status. Only `COMPLETED` / `PARTIAL_COMPLETE` jobs have a download URL.
total_items *	integer	Total item count in this job.
success_count *	integer	Number of items successfully scraped.
failed_count *	integer	Number of items that failed.
download_url	string	Signed URL to the merged `full_{job_id}.json` in GCS. Valid for 7 days; call this endpoint again to get a fresh URL. Returns `null` if the job has not yet completed.

JobRetrieveResponse

Field	Type	Description
job_id *	string (uuid)	The parent job identifier.
job_status *	string	Derived aggregate status across all batches: - `pending` — all batches are PENDING - `processing` — at least one batch is still PENDING - `completed` — all batches SUCCESS (terminal) - `failed` — all batches FAILED (terminal) - `partial_complete` — mix of SUCCESS and FAILED, no PENDING (terminal)
progress *	string	Human-readable progress, e.g. `17/47 batches complete`.
batches *	array<BatchDetail>	Per-batch detail, including signed download URL when available.

MerchantItemsSubmitRequest

Field	Type	Description
urls *	array<string>	List of Shopee merchant/shop page URLs to scrape.

PgntItemDetail

Field	Type	Description
url *	string	The submitted URL.
status *	string	Item status: `PENDING`, `SUCCESS`, or `FAILED`.
download_url	string	API endpoint to download the JSON result. Present only when `status == SUCCESS`.

PgntRetrieveResponse

Field	Type	Description
job_id *	string (uuid)	The job identifier.
task_type *	string	Type of task: `search`, `category`, or `merchant`.
status *	string	Job status: - `awaiting_approval` — waiting for Discord approval - `processing` — approved and running - `completed` — all URLs finished successfully - `partial_complete` — some URLs succeeded, some failed - `failed` — all URLs failed - `rejected` — approval denied
progress *	string	Human-readable progress, e.g. `3/5 URLs complete`.
items *	array<PgntItemDetail>	Per-URL detail with status and download URL when available.

PgntSubmitResponse

Field	Type	Description
job_id *	string (uuid)	Job identifier for polling via `GET /retrieve/{job_id}`.
task_type *	string	Type of task: `search`, `category`, or `merchant`.
status *	string	Initial status. `awaiting_approval` until approved via Discord.
cost *	integer	Credits deducted (6 credits per URL).
total_urls *	integer	Total number of URLs submitted.

QuotaErrorResponse

Field	Type	Description
detail *	string	Human-readable error message.
required *	integer	Credits required to process this submission.
current *	integer	User's current quota balance.
shortfall *	integer	How many additional credits the user needs (`required - current`).

SearchItemsSubmitRequest

Field	Type	Description
urls *	array<string>	List of Shopee search or category URLs to scrape.

ShopeeGetPcRetrieveResponse

Field	Type	Description
task_id *	integer
status *	string
result	object
refunded	integer
detail	string

ShopeeGetPcSubmitRequest

Field	Type	Description
region *	string	Shopee region code.
item_id *	string	Shopee item ID.
shop_id *	string	Shopee shop ID.

ShopeeGetPcSubmitResponse

Field	Type	Description
task_id *	integer
status *	string
cost *	integer
region *	string
item_id *	string
shop_id *	string

SubmitItem

Field	Type	Description
item_id *	string	The Shopee item ID.
shop_id *	string	The Shopee shop ID that owns this item.
country *	string	Two-letter country code. Must be one of: id, sg, th, ph, my, vn, tw.

SubmitRequest

Field	Type	Description
items *	array<SubmitItem>	Array of items to scrape. Between 1 and 50,000 entries.

SubmitResponse

Field	Type	Description
job_id *	string (uuid)	Parent identifier for the entire submission. Use this with `GET /retrieve/{job_id}`.
status *	string	Initial status. With the approval flow enabled this is `awaiting_approval` until a team member approves via Discord; afterwards it transitions to `processing`. With the approval flow disabled it starts as `initializing` and moves to `processing` within seconds. Rejected jobs end as `rejected` with credits refunded.
cost *	integer	Total credits deducted from the user's quota (10 credits per item_id).
refunded *	integer	Credits refunded immediately if any batches failed to be queued to RabbitMQ. Usually `0`; equal to `cost` if the broker was entirely unreachable.
total_items *	integer	Total number of items across all batches.
total_batches *	integer	Total number of batches (chunks of up to 50 items) generated.

ValidationError

Field	Type	Description
loc *	array<string \| integer>
msg *	string
type *	string