Skip to content

Structured Utilities data: send urls. Cursor-paginated, 1 credit per successful call.

GET · POST1 credit / successful callCursor-paginated
GEThttps://api.scrapier.io/api/v2/website-content-crawler/Open in Playground

Call GET /api/v2/website-content-crawler/ with urls. The results are in data.pages; send data.next_cursor back as cursor to get the next page. 17 optional parameters refine the call (see Parameters).

Terminal window
curl -G "https://api.scrapier.io/api/v2/website-content-crawler/" \
-H "X-API-Key: YOUR_API_KEY" \
--data-urlencode "urls=https://docs.apify.com/academy/scraping-basics-javascript" \
--max-time 210
NameTypeRequiredDefaultDescription
X-API-Key headerstringyes—Your API key from the dashboard.
urlsstringyese.g. https://docs.apify.com/academy/scraping-basics-javascriptStart URL of the site to crawl (one per walk). Required first call.
cursorstringno—Pagination cursor returned in the previous response's next_cursor. Pass it back unchanged to fetch the next page. When next_cursor is null, you have reached the last page.
page_sizeintegerno10Pages returned per call.
locked in cursor
max_total_itemsintegerno100Pages returned over the whole walk.
locked in cursor
max_crawl_depthintegerno1Link hops from the start URL (0 = start URL only).
locked in cursor
max_crawl_pagesintegerno—Cap on pages fetched over the whole walk.
locked in cursor
include_url_globsstringno—fnmatch patterns a followed URL must match.
locked in cursor
exclude_url_globsstringno—fnmatch patterns that exclude a URL.
locked in cursor
save_markdownbooleannotrueFill the markdown field.
locked in cursor
include_htmlbooleannofalseAdd readability HTML (large).
locked in cursor
remove_elements_css_selectorstringno—CSS selectors removed before text extraction.
locked in cursor
readable_text_char_thresholdintegerno100Minimum text length for a page to count.
locked in cursor
keep_url_fragmentsbooleannofalse
locked in cursor
ignore_canonical_urlbooleannofalse
locked in cursor
custom_http_headersstringno—JSON object of extra request headers.
locked in cursor
fingerprint_profilestringnochrome_windows
chrome_windowschrome_macosfirefox_windows
locked in cursor
browser_fallbackbooleannotrueBrowser escalation for bot walls.
locked in cursor
solve_captchasbooleannotrueAllow the paid CapSolver tier.
locked in cursor
Advanced
NameTypeDefaultDescription
use_proxybooleantrueRoute the request through Scrapier's residential proxy pool. Default: true. Leave as is unless support tells you otherwise.
raw_databooleanfalseWhen true, the response carries the full raw upstream payload(s) the scraper received (GraphQL / JSON API bodies, HTML, embedded JSON-LD -- exactly as the platform returned them) under raw_responses, in place of the formatted items. When false (default) you get the formatted output. Pagination (cursor / next_cursor) works the same in both modes.
{
"success": true,
"data": {
"url": "https://docs.apify.com/academy/scraping-basics-javascript",
"page": 1,
"page_size": 10,
"pages_crawled_total": 10,
"pages": [
{
"input_url": "https://docs.apify.com/academy/scraping-basics-javascript",
"url": "https://docs.apify.com/academy/scraping-basics-javascript",
"crawl": {
"loaded_url": "https://docs.apify.com/academy/scraping-basics-javascript",
"loaded_time": "2026-10-19T12:06:41.301Z",
"referrer_url": "https://docs.apify.com/academy/scraping-basics-javascript",
"http_status_code": 200,
"depth": 0,
"content_length_bytes": 148211,
"content_type": "text/html; charset=utf-8"
},
"metadata": {
"title": "Web scraping basics for JavaScript devs | Apify Documentation",
"description": null,
"author": null,
"keywords": null,
"language_code": "en",
"json_ld": null,
"headers": {}
},
"text": "Web scraping basics ...",
"markdown": "# Web scraping basics ...",
"html": null,
"debug": {
"request_handler_mode": "http",
"page_type": "Article",
"total_links": 112,
"antibot": {
"tier": "http",
"vendor": "",
"solved_with": [],
"attempts": 1
},
"lengths": {
"text": 6362,
"markdown": 6890
}
},
"scraped_at": "2026-10-19T12:06:41.301Z",
"position": 1,
"page": 1
}
],
"warnings": [],
"next_cursor": "zc1.eNpVjksOgjAQhq9..."
}
}

Sample from the live catalog; arrays trimmed to 3 items. Results are in data.pages.

FieldTypeExampleDescription
urlstringhttps://docs.apify.com/academy/scraping-basic…—
pageinteger1—
page_sizeinteger10—
pages_crawled_totalinteger10—
pagesarray[1 item]The results.
warningsarray[]—
next_cursorstringzc1.eNpVjksOgjAQhq9...Pass back as cursor for the next page. null on the last page.
pages[].input_urlstringhttps://docs.apify.com/academy/scraping-basic…—
pages[].urlstringhttps://docs.apify.com/academy/scraping-basic…—
pages[].crawlobject{…}—
pages[].metadataobject{…}—
pages[].textstringWeb scraping basics ...—
pages[].markdownstring# Web scraping basics ...—
pages[].htmlnullnull—
pages[].debugobject{…}—
pages[].scraped_atstring2026-10-19T12:06:41.301Z—
pages[].positioninteger1—
pages[].pageinteger1—

Pass data.next_cursor back as cursor until it is null. Required inputs are ignored once a cursor is sent. Each page is one billed call. More in Pagination.

const url = "https://api.scrapier.io/api/v2/website-content-crawler/";
let params = { urls: "https://docs.apify.com/academy/scraping-basics-javascript" };
const items = [];
for (let page = 0; page < 10; page++) { // each page is one billed call
const res = await fetch(`${url}?${new URLSearchParams(params)}`, {
headers: { "X-API-Key": "YOUR_API_KEY" },
signal: AbortSignal.timeout(210_000),
});
const body = await res.json();
if (!res.ok) throw new Error(body.error ?? body.detail ?? `HTTP ${res.status}`);
const data = body.data;
items.push(...(data.pages ?? []));
if (!data.next_cursor) break; // null = last page
params = { cursor: data.next_cursor };
}

Failed calls are free. Branch on the HTTP status and read the message from error (or detail). Full details in Errors.

StatusMeaningRetry?Charged?
400Bad or missing input.No — fix the paramsNo
401 / 403Missing, invalid or revoked API key.No — fix the keyNo
402Out of credits (body has balance).No — top upNo
404Unknown endpoint slug.No — check the slugNo
429 / 503Busy or throttled.Yes — after Retry-AfterNo
5xxTarget site or upstream failure, or timeout.Yes — with backoffNo
  • Locked into the cursor: page_size, max_total_items, max_crawl_depth, max_crawl_pages, include_url_globs, exclude_url_globs, save_markdown, include_html, remove_elements_css_selector, readable_text_char_threshold, keep_url_fragments, ignore_canonical_url, custom_http_headers, fingerprint_profile, browser_fallback, solve_captchas. Set them on the first call; later pages reuse them.
  • Contact Extractor — Structured Utilities data: send urls. Cursor-paginated, 2 credits per successful call.
  • Industry Email Scraper — Structured Utilities data: send queries and location. Cursor-paginated, 2 credits per successful call.
  • Social Media Platform Email Scraper — Structured Utilities data: send keywords. Cursor-paginated, 1 credit per successful call.