Does the Markdown include navigation and footers?

Crawl results are the whole page, so yes. Strip the repeated navigation before chunking. When you scrape or batch scrape, extractContentOnly drops navigation, headers, footers and sidebars for you.

How big a site can I crawl?

Up to your plan's page limit per crawl: 100 on Free, 1,000 on Starter, 5,000 on Builder and 10,000 on Pro. The first 100 pages come back with the crawl, and you read the rest through the paged results endpoint.

Should I map or crawl?

Map when you want to choose pages yourself: it lists a site's URLs in seconds for 1 credit, and you batch scrape the ones you pick. Crawl when you'd rather describe the pages and let Spidra find them.

Docs into RAG

Docs and sites into Markdown your model can read

Crawl a documentation site, or map it and pick the pages, and get clean Markdown with code blocks intact. Or give your agent the same tools through the MCP server.

You send a start page

docs.python.org/3/library/asyncio.html

Instruction: Individual Python asyncio reference pages

Crawl

You get

5 pages · 5 credits

asyncio-runner.html

docs.python.org/3/library

10,407 characters2 code blocks

asyncio-task.html

docs.python.org/3/library

59,841 characters32 code blocks

asyncio-stream.html

docs.python.org/3/library

22,038 characters11 code blocks

2 more, Markdown with code fences intact

Docs to chunks

From a docs site to chunks you can cite

Map the site, pick the pages, read them as Markdown and chunk on headings. Real numbers from FastAPI's and Python's docs.

Map

619 URLs found on fastapi.tiangolo.com

1 credit

Pick

36 matched "dependencies"; drop /de/, /es/, /fr/

free

Read

Each page as Markdown, code fenced

1 credit a page

Chunk

Split on headings, keep the URL for citations

your code

A chunk that keeps its code and its source

From a real scrape of Python's asyncio docs: the code example arrives as a fenced block, so a chunker can keep it whole, and every chunk can point back to the page it came from.

one chunk

{
  "source": "https://docs.python.org/3/library/asyncio-queue.html",
  "title": "Queues",
  "text": "## Examples\n\nQueues can be used to distribute workload between several concurrent tasks:\n\n```\nimport asyncio\nimport random\nimport time\n\nasync def worker(name, queue):\n    while True:\n…"
}

How it works

Pick the pages, then read them

Map shows what a site has in seconds. Crawl reads the pages you describe. MCP hands both to your agent.

map_site.py

1import os
2from spidra import Spidra
3 
4client = Spidra(api_key=os.environ["SPIDRA_API_KEY"])
5 
6site = client.map("https://fastapi.tiangolo.com/", search="dependencies", limit=10)
7 
8print(site.total, "URLs matched,", site.scanned, "scanned")
9for link in site.links[:5]:
10    print(link.url)

Docs sites often have translated copies. Filter paths like /de/ and /es/ before you scrape.

Plan it

What it costs, and what to know first

A 500-page documentation site, roughly

Map the site1 credit
Read 500 pages as Markdown, no AI500 credits
Read it again each month to stay currentabout 500 credits a month

Inside the Starter plan's 5,000 credits a month, which also allows 1,000 pages per crawl.

Know before you build

Crawled Markdown includes the menus
Strip repeated navigation before chunking, or batch scrape with extractContentOnly.
Markdown links expire
Crawl results link to each page's Markdown for about an hour. Download it right away.
Translations come along
Docs sites list translated copies. Filter paths like /de/ and /es/.
Pages per crawl follow your plan
From 100 on Free to 10,000 on Pro. Map lists up to 5,000 URLs a call on any plan.

Copy and paste

Fine for ten pages, out of date the week after.

A homegrown crawler

Works until the docs move to a new theme or start rendering with JavaScript.

Spidra

Map, crawl or batch into clean Markdown with code intact, for 1 credit a page and no AI.

What you get

Pages ready for chunking and citing

Every page keeps its URL and title, so each chunk can point back to where it came from.

A page from a crawl

urlstringThe page's address, your citation
titlestringThe page title
markdownstringA link to the page as Markdown, fenced code included
dataobjectFields from your schema, when you give one
depthnumberHow many links from the start page
parentUrlstringThe page it was found on

Crawled Markdown is the whole page, navigation included, so strip repeated menus before chunking. Scrape and batch can drop them for you.

Where it works

Documentation sites

Docs built with Docusaurus, MkDocs, Sphinx and the rest, with code blocks kept as code.

Help centers and blogs

Articles crawled by description, like "Individual blog posts".

Research papers

arXiv and PubMed results through search, and PDFs through Parse.

GitHub issues and PRs

Search issues and pull requests with GitHub's own qualifiers.

FAQ

Questions, answered

AI agents and RAG

Claude Code, Claude Desktop, Cursor, Windsurf, VS Code and any client that speaks MCP. Use the hosted server at https://mcp.spidra.io/mcp and sign in, or run it locally with npx -y spidra-mcp.