Use these docs with AI
These docs are built to be read by LLMs and agents as well as people. Every page has a clean Markdown version, and there are index files that give an agent the whole API in one fetch.
llms.txt files
Section titled “llms.txt files”| File | What’s in it |
|---|---|
/llms.txt |
The index: short agent instructions (base URL, auth, pagination, billing, errors) and a link to every page’s Markdown. Start here. |
/llms-full.txt |
Every page in one file, in sidebar order. Good for pasting into a long-context model. |
/llms-{platform}.txt |
One file per platform, for every platform, e.g. /llms-instagram.txt. Every endpoint with its parameters, types, required flags and a cURL line. |
Every platform file lists each endpoint with its parameters, price and a cURL line, straight from the live catalog, so new endpoints show up without any manual step. Platforms with hand-written descriptions, use cases and tips add them to their file: Instagram, Google and Amazon. The full list of platforms is on All platforms.
Markdown for every page
Section titled “Markdown for every page”Add .md to any page URL to get its Markdown:
https://docs.scrapier.io/quickstart/→https://docs.scrapier.io/quickstart.mdhttps://docs.scrapier.io/→https://docs.scrapier.io/index.md
You can also ask for Markdown with the Accept header, and keep the normal URL:
curl -H 'Accept: text/markdown' https://docs.scrapier.io/quickstart/Each HTML page also links its Markdown version with <link rel="alternate" type="text/markdown">.
The page menu
Section titled “The page menu”Every page has a Copy page button next to its title. The menu beside it has:
- View as Markdown: opens the
.mdversion. - Open in ChatGPT / Claude / Perplexity / Cursor: starts a chat that reads this page. On endpoint pages, the prompt describes the endpoint and its parameters, and is also copied to your clipboard.
- Copy MCP server URL and Connect to Cursor / VS Code: install the MCP server in one click.
Agent skills discovery
Section titled “Agent skills discovery”https://mcp.scrapier.io/.well-known/agent-skills/index.json lists the agent skills for tools that discover skills automatically, such as npx skills add https://mcp.scrapier.io. The same path on this site redirects there.
Crawlers
Section titled “Crawlers”AI crawlers and assistants (GPTBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, OAI-SearchBot and others) are allowed to read everything. See /robots.txt. The Markdown files are marked noindex for search engines so they don’t compete with the HTML pages.
For agents: the short version
Section titled “For agents: the short version”- Start at https://docs.scrapier.io/llms.txt.
- Never invent endpoints or parameters. Read them from
GET https://api.scrapier.io/api/v2/endpoints(free, no key) or the platform’s llms file. - Prefer the MCP server when your client supports it.