Rate limits & reliability
Scrapier doesn’t publish a fixed per-key request quota. Capacity is shared, so the API tells you when to slow down, and your client should listen.
Back off on 429 and 503
Section titled “Back off on 429 and 503”429means you’re being throttled. Wait for the number of seconds in theRetry-Afterheader, then retry.503means Scrapier is at capacity right now. It usually carriesRetry-After: 30. Wait, then retry.- If there’s no
Retry-After, use exponential backoff with jitter: wait about 1 s, then 2 s, then 4 s and so on, each multiplied by a random factor, up to a minute.
Both are free: failed calls never cost credits. The Errors page has a ready-made retry snippet.
Keep concurrency modest
Section titled “Keep concurrency modest”Run no more than 3 requests in parallel per key. If you scrape a long list, work through it with a small pool rather than firing everything at once. You’ll see fewer 429 and 503 responses, and the whole job usually finishes sooner.
Timeouts
Section titled “Timeouts”Scraping a live website takes time. Some endpoints take several minutes.
- Set your HTTP client’s timeout to at least 210 seconds:
curl --max-time 210,AbortSignal.timeout(210_000)in JavaScript,timeout=210in Pythonrequests. - Automation tools often default to much shorter timeouts. Raise them in n8n, Make or Zapier where you can.
- A timeout on our side returns
502or504. Both are free and safe to retry.
Retrying safely
Section titled “Retrying safely”5xxand429: retry with backoff. You’re only charged when a call succeeds.400,401,403,404: don’t retry. Fix the request or key first.402with abalancefield: don’t retry. Top up. A402withoutbalancecame from the target site; treat it like a5xx.- GET requests are safe to repeat. Keep in mind that every successful repeat is billed again, so don’t re-run calls that already succeeded.