> Documentation index: https://docs.scrapier.io/llms.txt
> Auth: header `X-API-Key: <key>` · Base: https://api.scrapier.io/api/v2 · Synchronous; paginate with `cursor` ← `data.next_cursor`.

# Website Content Crawler

Structured Utilities data: send `urls`. Cursor-paginated, 1 credit per successful call.

`GET|POST https://api.scrapier.io/api/v2/website-content-crawler/` · 1 credit per successful call · Cursor-paginated

## Overview

Call `GET /api/v2/website-content-crawler/` with `urls`. The results are in `data.pages`; send `data.next_cursor` back as `cursor` to get the next page. 17 optional parameters refine the call (see Parameters).

## Request

```bash
curl -G "https://api.scrapier.io/api/v2/website-content-crawler/" \
  -H "X-API-Key: YOUR_API_KEY" \
  --data-urlencode "urls=https://docs.apify.com/academy/scraping-basics-javascript" \
  --max-time 210
```

```python
import requests

resp = requests.get(
    "https://api.scrapier.io/api/v2/website-content-crawler/",
    headers={"X-API-Key": "YOUR_API_KEY"},
    params={"urls": "https://docs.apify.com/academy/scraping-basics-javascript"},
    timeout=210,
)
body = resp.json()
if not resp.ok:
    raise RuntimeError(body.get("error") or body.get("detail"))

print(body["data"]["pages"])
```

POST with a JSON body works too (same parameters; the body is merged over the query string).

## Parameters

| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| `X-API-Key` (header) | string | yes | — | Your API key. |
| `urls` | string | yes | e.g. `https://docs.apify.com/academy/scraping-basics-javascript` | Start URL of the site to crawl (one per walk). Required first call. |
| `cursor` | string | no | — | Pagination cursor returned in the previous response's `next_cursor`. Pass it back unchanged to fetch the next page. When `next_cursor` is null, you have reached the last page. |
| `page_size` | integer | no | `10` | Pages returned per call. Locked in cursor. |
| `max_total_items` | integer | no | `100` | Pages returned over the whole walk. Locked in cursor. |
| `max_crawl_depth` | integer | no | `1` | Link hops from the start URL (0 = start URL only). Locked in cursor. |
| `max_crawl_pages` | integer | no | — | Cap on pages fetched over the whole walk. Locked in cursor. |
| `include_url_globs` | string | no | — | fnmatch patterns a followed URL must match. Locked in cursor. |
| `exclude_url_globs` | string | no | — | fnmatch patterns that exclude a URL. Locked in cursor. |
| `save_markdown` | boolean | no | `true` | Fill the markdown field. Locked in cursor. |
| `include_html` | boolean | no | `false` | Add readability HTML (large). Locked in cursor. |
| `remove_elements_css_selector` | string | no | — | CSS selectors removed before text extraction. Locked in cursor. |
| `readable_text_char_threshold` | integer | no | `100` | Minimum text length for a page to count. Locked in cursor. |
| `keep_url_fragments` | boolean | no | `false` | Locked in cursor. |
| `ignore_canonical_url` | boolean | no | `false` | Locked in cursor. |
| `custom_http_headers` | string | no | — | JSON object of extra request headers. Locked in cursor. |
| `fingerprint_profile` | string | no | `chrome_windows` | One of: `chrome_windows`, `chrome_macos`, `firefox_windows`. Locked in cursor. |
| `browser_fallback` | boolean | no | `true` | Browser escalation for bot walls. Locked in cursor. |
| `solve_captchas` | boolean | no | `true` | Allow the paid CapSolver tier. Locked in cursor. |
| `use_proxy` | boolean | no | `true` | Route the request through Scrapier's residential proxy pool. Default: `true`. Leave as is unless support tells you otherwise. |
| `raw_data` | boolean | no | `false` | When true, the response carries the full raw upstream payload(s) the scraper received (GraphQL / JSON API bodies, HTML, embedded JSON-LD -- exactly as the platform returned them) under `raw_responses`, in place of the formatted items. When false (default) you get the formatted output. Pagination (`cursor` / `next_cursor`) works the same in both modes. |

## Response

HTTP 200. Results are in `data.pages`. Sample from the live catalog, arrays trimmed to 3 items.

```json
{
  "success": true,
  "data": {
    "url": "https://docs.apify.com/academy/scraping-basics-javascript",
    "page": 1,
    "page_size": 10,
    "pages_crawled_total": 10,
    "pages": [
      {
        "input_url": "https://docs.apify.com/academy/scraping-basics-javascript",
        "url": "https://docs.apify.com/academy/scraping-basics-javascript",
        "crawl": {
          "loaded_url": "https://docs.apify.com/academy/scraping-basics-javascript",
          "loaded_time": "2026-10-19T12:06:41.301Z",
          "referrer_url": "https://docs.apify.com/academy/scraping-basics-javascript",
          "http_status_code": 200,
          "depth": 0,
          "content_length_bytes": 148211,
          "content_type": "text/html; charset=utf-8"
        },
        "metadata": {
          "title": "Web scraping basics for JavaScript devs | Apify Documentation",
          "description": null,
          "author": null,
          "keywords": null,
          "language_code": "en",
          "json_ld": null,
          "headers": {}
        },
        "text": "Web scraping basics ...",
        "markdown": "# Web scraping basics ...",
        "html": null,
        "debug": {
          "request_handler_mode": "http",
          "page_type": "Article",
          "total_links": 112,
          "antibot": {
            "tier": "http",
            "vendor": "",
            "solved_with": [],
            "attempts": 1
          },
          "lengths": {
            "text": 6362,
            "markdown": 6890
          }
        },
        "scraped_at": "2026-10-19T12:06:41.301Z",
        "position": 1,
        "page": 1
      }
    ],
    "warnings": [],
    "next_cursor": "zc1.eNpVjksOgjAQhq9..."
  }
}
```

## Response fields

| Field | Type | Example | Description |
|---|---|---|---|
| `url` | string | `https://docs.apify.com/academy/scraping-basic…` |  |
| `page` | integer | `1` |  |
| `page_size` | integer | `10` |  |
| `pages_crawled_total` | integer | `10` |  |
| `pages` | array | `[1 item]` | The results. |
| `warnings` | array | `[]` |  |
| `next_cursor` | string | `zc1.eNpVjksOgjAQhq9...` | Pass back as `cursor` for the next page. `null` on the last page. |
| `pages[].input_url` | string | `https://docs.apify.com/academy/scraping-basic…` |  |
| `pages[].url` | string | `https://docs.apify.com/academy/scraping-basic…` |  |
| `pages[].crawl` | object | `{…}` |  |
| `pages[].metadata` | object | `{…}` |  |
| `pages[].text` | string | `Web scraping basics ...` |  |
| `pages[].markdown` | string | `# Web scraping basics ...` |  |
| `pages[].html` | null | `null` |  |
| `pages[].debug` | object | `{…}` |  |
| `pages[].scraped_at` | string | `2026-10-19T12:06:41.301Z` |  |
| `pages[].position` | integer | `1` |  |
| `pages[].page` | integer | `1` |  |

## Pagination

Pass `data.next_cursor` back as `cursor` until it is `null`. Required inputs are ignored once a cursor is sent. Each page is one billed call.

```python
import requests

URL = "https://api.scrapier.io/api/v2/website-content-crawler/"
HEADERS = {"X-API-Key": "YOUR_API_KEY"}
params = {"urls": "https://docs.apify.com/academy/scraping-basics-javascript"}
items = []

for _ in range(10):  # each page is one billed call
    resp = requests.get(URL, headers=HEADERS, params=params, timeout=210)
    resp.raise_for_status()
    data = resp.json()["data"]
    items.extend(data.get("pages") or [])
    if not data.get("next_cursor"):  # None = last page
        break
    params = {"cursor": data["next_cursor"]}
```

## Errors

Failed calls are free. Branch on HTTP status; the message is in `error` (or `detail`). See https://docs.scrapier.io/errors.md

- **400** Bad input. The exact message varies by endpoint. `{"success":false,"error":"…"}`
- **402** Balance below the endpoint's price. `{"error":"Insufficient credits. Buy more to scrape more.","balance":1,"required":2}`
- **403** Invalid or unknown API key. `{"detail":"Authentication service unavailable or invalid API key."}`
- **429** Throttled — wait for Retry-After. `{"detail":"Request was throttled. Expected available in N seconds."}`
- **404** unknown slug · **503** busy (retry after `Retry-After`) · **5xx** upstream failure or timeout (retry with backoff).

## Notes

- Locked into the cursor: `page_size`, `max_total_items`, `max_crawl_depth`, `max_crawl_pages`, `include_url_globs`, `exclude_url_globs`, `save_markdown`, `include_html`, `remove_elements_css_selector`, `readable_text_char_threshold`, `keep_url_fragments`, `ignore_canonical_url`, `custom_http_headers`, `fingerprint_profile`, `browser_fallback`, `solve_captchas`. Set them on the first call; later pages reuse them.

## Related

- [Contact Extractor](https://docs.scrapier.io/api-reference/utilities/contact-extractor.md): Structured Utilities data: send `urls`. Cursor-paginated, 2 credits per successful call.
- [Industry Email Scraper](https://docs.scrapier.io/api-reference/utilities/industry-email.md): Structured Utilities data: send `queries` and `location`. Cursor-paginated, 2 credits per successful call.
- [Social Media Platform Email Scraper](https://docs.scrapier.io/api-reference/utilities/social-media-platform-email.md): Structured Utilities data: send `keywords`. Cursor-paginated, 1 credit per successful call.
