Skip to content

These docs are built to be read by LLMs and agents as well as people. Every page has a clean Markdown version, and there are index files that give an agent the whole API in one fetch.

File What’s in it
/llms.txt The index: short agent instructions (base URL, auth, pagination, billing, errors) and a link to every page’s Markdown. Start here.
/llms-full.txt Every page in one file, in sidebar order. Good for pasting into a long-context model.
/llms-{platform}.txt One file per platform, for every platform, e.g. /llms-instagram.txt. Every endpoint with its parameters, types, required flags and a cURL line.

Every platform file lists each endpoint with its parameters, price and a cURL line, straight from the live catalog, so new endpoints show up without any manual step. Platforms with hand-written descriptions, use cases and tips add them to their file: Instagram, Google and Amazon. The full list of platforms is on All platforms.

Add .md to any page URL to get its Markdown:

  • https://docs.scrapier.io/quickstart/ → https://docs.scrapier.io/quickstart.md
  • https://docs.scrapier.io/ → https://docs.scrapier.io/index.md

You can also ask for Markdown with the Accept header, and keep the normal URL:

Terminal window
curl -H 'Accept: text/markdown' https://docs.scrapier.io/quickstart/

Each HTML page also links its Markdown version with <link rel="alternate" type="text/markdown">.

Every page has a Copy page button next to its title. The menu beside it has:

  • View as Markdown: opens the .md version.
  • Open in ChatGPT / Claude / Perplexity / Cursor: starts a chat that reads this page. On endpoint pages, the prompt describes the endpoint and its parameters, and is also copied to your clipboard.
  • Copy MCP server URL and Connect to Cursor / VS Code: install the MCP server in one click.

https://mcp.scrapier.io/.well-known/agent-skills/index.json lists the agent skills for tools that discover skills automatically, such as npx skills add https://mcp.scrapier.io. The same path on this site redirects there.

AI crawlers and assistants (GPTBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, OAI-SearchBot and others) are allowed to read everything. See /robots.txt. The Markdown files are marked noindex for search engines so they don’t compete with the HTML pages.

  • Start at https://docs.scrapier.io/llms.txt.
  • Never invent endpoints or parameters. Read them from GET https://api.scrapier.io/api/v2/endpoints (free, no key) or the platform’s llms file.
  • Prefer the MCP server when your client supports it.