Website Content Crawler
Structured Utilities data: send urls. Cursor-paginated, 1 credit per successful call.
Overview
Section titled “Overview”Call GET /api/v2/website-content-crawler/ with urls. The results are in data.pages; send data.next_cursor back as cursor to get the next page. 17 optional parameters refine the call (see Parameters).
Request
Section titled “Request”curl -G "https://api.scrapier.io/api/v2/website-content-crawler/" \ -H "X-API-Key: YOUR_API_KEY" \ --data-urlencode "urls=https://docs.apify.com/academy/scraping-basics-javascript" \ --max-time 210const params = new URLSearchParams({ urls: "https://docs.apify.com/academy/scraping-basics-javascript" });
const res = await fetch(`https://api.scrapier.io/api/v2/website-content-crawler/?${params}`, { headers: { "X-API-Key": "YOUR_API_KEY" }, signal: AbortSignal.timeout(210_000),});const body = await res.json();if (!res.ok) throw new Error(body.error ?? body.detail ?? `HTTP ${res.status}`);
console.log(body.data.pages);import requests
resp = requests.get( "https://api.scrapier.io/api/v2/website-content-crawler/", headers={"X-API-Key": "YOUR_API_KEY"}, params={"urls": "https://docs.apify.com/academy/scraping-basics-javascript"}, timeout=210,)body = resp.json()if not resp.ok: raise RuntimeError(body.get("error") or body.get("detail"))
print(body["data"]["pages"])curl -X POST "https://api.scrapier.io/api/v2/website-content-crawler/" \ -H "X-API-Key: YOUR_API_KEY" \ -H "Content-Type: application/json" \ -d '{"urls":"https://docs.apify.com/academy/scraping-basics-javascript"}' \ --max-time 210Parameters
Section titled “Parameters”| Name | Type | Required | Default | Description |
|---|---|---|---|---|
X-API-Key header | string | yes | — | Your API key from the dashboard. |
urls | string | yes | e.g. https://docs.apify.com/academy/scraping-basics-javascript | Start URL of the site to crawl (one per walk). Required first call. |
cursor | string | no | — | Pagination cursor returned in the previous response's next_cursor. Pass it back unchanged to fetch the next page. When next_cursor is null, you have reached the last page. |
page_size | integer | no | 10 | Pages returned per call. locked in cursor |
max_total_items | integer | no | 100 | Pages returned over the whole walk. locked in cursor |
max_crawl_depth | integer | no | 1 | Link hops from the start URL (0 = start URL only). locked in cursor |
max_crawl_pages | integer | no | — | Cap on pages fetched over the whole walk. locked in cursor |
include_url_globs | string | no | — | fnmatch patterns a followed URL must match. locked in cursor |
exclude_url_globs | string | no | — | fnmatch patterns that exclude a URL. locked in cursor |
save_markdown | boolean | no | true | Fill the markdown field. locked in cursor |
include_html | boolean | no | false | Add readability HTML (large). locked in cursor |
remove_elements_css_selector | string | no | — | CSS selectors removed before text extraction. locked in cursor |
readable_text_char_threshold | integer | no | 100 | Minimum text length for a page to count. locked in cursor |
keep_url_fragments | boolean | no | false | locked in cursor |
ignore_canonical_url | boolean | no | false | locked in cursor |
custom_http_headers | string | no | — | JSON object of extra request headers. locked in cursor |
fingerprint_profile | string | no | chrome_windows | chrome_windowschrome_macosfirefox_windows locked in cursor |
browser_fallback | boolean | no | true | Browser escalation for bot walls. locked in cursor |
solve_captchas | boolean | no | true | Allow the paid CapSolver tier. locked in cursor |
Advanced
| Name | Type | Default | Description |
|---|---|---|---|
use_proxy | boolean | true | Route the request through Scrapier's residential proxy pool. Default: true. Leave as is unless support tells you otherwise. |
raw_data | boolean | false | When true, the response carries the full raw upstream payload(s) the scraper received (GraphQL / JSON API bodies, HTML, embedded JSON-LD -- exactly as the platform returned them) under raw_responses, in place of the formatted items. When false (default) you get the formatted output. Pagination (cursor / next_cursor) works the same in both modes. |
Response
Section titled “Response”{ "success": true, "data": { "url": "https://docs.apify.com/academy/scraping-basics-javascript", "page": 1, "page_size": 10, "pages_crawled_total": 10, "pages": [ { "input_url": "https://docs.apify.com/academy/scraping-basics-javascript", "url": "https://docs.apify.com/academy/scraping-basics-javascript", "crawl": { "loaded_url": "https://docs.apify.com/academy/scraping-basics-javascript", "loaded_time": "2026-10-19T12:06:41.301Z", "referrer_url": "https://docs.apify.com/academy/scraping-basics-javascript", "http_status_code": 200, "depth": 0, "content_length_bytes": 148211, "content_type": "text/html; charset=utf-8" }, "metadata": { "title": "Web scraping basics for JavaScript devs | Apify Documentation", "description": null, "author": null, "keywords": null, "language_code": "en", "json_ld": null, "headers": {} }, "text": "Web scraping basics ...", "markdown": "# Web scraping basics ...", "html": null, "debug": { "request_handler_mode": "http", "page_type": "Article", "total_links": 112, "antibot": { "tier": "http", "vendor": "", "solved_with": [], "attempts": 1 }, "lengths": { "text": 6362, "markdown": 6890 } }, "scraped_at": "2026-10-19T12:06:41.301Z", "position": 1, "page": 1 } ], "warnings": [], "next_cursor": "zc1.eNpVjksOgjAQhq9..." }}Bad input. The exact message varies by endpoint.
{ "success": false, "error": "…"}Balance below the endpoint’s price.
{ "error": "Insufficient credits. Buy more to scrape more.", "balance": 1, "required": 2}Invalid or unknown API key.
{ "detail": "Authentication service unavailable or invalid API key."}Throttled — wait for Retry-After.
{ "detail": "Request was throttled. Expected available in N seconds."}Sample from the live catalog; arrays trimmed to 3 items. Results are in data.pages.
Response fields
Section titled “Response fields”| Field | Type | Example | Description |
|---|---|---|---|
url | string | https://docs.apify.com/academy/scraping-basic… | — |
page | integer | 1 | — |
page_size | integer | 10 | — |
pages_crawled_total | integer | 10 | — |
pages | array | [1 item] | The results. |
warnings | array | [] | — |
next_cursor | string | zc1.eNpVjksOgjAQhq9... | Pass back as cursor for the next page. null on the last page. |
pages[].input_url | string | https://docs.apify.com/academy/scraping-basic… | — |
pages[].url | string | https://docs.apify.com/academy/scraping-basic… | — |
pages[].crawl | object | {…} | — |
pages[].metadata | object | {…} | — |
pages[].text | string | Web scraping basics ... | — |
pages[].markdown | string | # Web scraping basics ... | — |
pages[].html | null | null | — |
pages[].debug | object | {…} | — |
pages[].scraped_at | string | 2026-10-19T12:06:41.301Z | — |
pages[].position | integer | 1 | — |
pages[].page | integer | 1 | — |
Pagination
Section titled “Pagination”Pass data.next_cursor back as cursor until it is null. Required inputs are ignored once a cursor is sent. Each page is one billed call. More in Pagination.
const url = "https://api.scrapier.io/api/v2/website-content-crawler/";let params = { urls: "https://docs.apify.com/academy/scraping-basics-javascript" };const items = [];
for (let page = 0; page < 10; page++) { // each page is one billed call const res = await fetch(`${url}?${new URLSearchParams(params)}`, { headers: { "X-API-Key": "YOUR_API_KEY" }, signal: AbortSignal.timeout(210_000), }); const body = await res.json(); if (!res.ok) throw new Error(body.error ?? body.detail ?? `HTTP ${res.status}`); const data = body.data; items.push(...(data.pages ?? [])); if (!data.next_cursor) break; // null = last page params = { cursor: data.next_cursor };}import requests
URL = "https://api.scrapier.io/api/v2/website-content-crawler/"HEADERS = {"X-API-Key": "YOUR_API_KEY"}params = {"urls": "https://docs.apify.com/academy/scraping-basics-javascript"}items = []
for _ in range(10): # each page is one billed call resp = requests.get(URL, headers=HEADERS, params=params, timeout=210) resp.raise_for_status() data = resp.json()["data"] items.extend(data.get("pages") or []) if not data.get("next_cursor"): # None = last page break params = {"cursor": data["next_cursor"]}Errors
Section titled “Errors”Failed calls are free. Branch on the HTTP status and read the message from error (or detail). Full details in Errors.
| Status | Meaning | Retry? | Charged? |
|---|---|---|---|
400 | Bad or missing input. | No — fix the params | No |
401 / 403 | Missing, invalid or revoked API key. | No — fix the key | No |
402 | Out of credits (body has balance). | No — top up | No |
404 | Unknown endpoint slug. | No — check the slug | No |
429 / 503 | Busy or throttled. | Yes — after Retry-After | No |
5xx | Target site or upstream failure, or timeout. | Yes — with backoff | No |
- Locked into the cursor:
page_size,max_total_items,max_crawl_depth,max_crawl_pages,include_url_globs,exclude_url_globs,save_markdown,include_html,remove_elements_css_selector,readable_text_char_threshold,keep_url_fragments,ignore_canonical_url,custom_http_headers,fingerprint_profile,browser_fallback,solve_captchas. Set them on the first call; later pages reuse them.
Related
Section titled “Related”- Contact Extractor — Structured Utilities data: send
urls. Cursor-paginated, 2 credits per successful call. - Industry Email Scraper — Structured Utilities data: send
queriesandlocation. Cursor-paginated, 2 credits per successful call. - Social Media Platform Email Scraper — Structured Utilities data: send
keywords. Cursor-paginated, 1 credit per successful call.