A crawler your agents can drive.
OpenCrawl is a fast, open crawl engine with an MCP server built in. Your agents call it like any other tool. Your pipelines read it like any other data source.
Request an invite- Pages crawled
- 12,480
- +4.3%vs last crawl
- 9,210 to 12,480 over the last 14 crawls
- Fetched without error
- 97.4%
- +1.6 ptsvs last crawl
- 93.1% to 97.4% over the last 14 crawls
- Median response
- 182ms
- −15.0%vs last crawl
- 268ms to 182ms over the last 14 crawls
- Pages a second
- 208
- +11.8%vs last crawl
- 151 to 208 over the last 14 crawls
Crawling as a tool call.
opencrawl mcp serves the engine over the Model Context Protocol. Any MCP client can start a crawl, ask for its progress, and query the pages it found by path, status or content, then reason over real responses rather than guesses.
Long crawls run in the background and are saved, so an agent can pick up a result hours later, or compare it with last week’s.
you › Crawl our docs and list every page that links somewhere broken.
→ run_crawl{ url: "https://example.com/docs/", max_pages: 2000 }
→ query_pages{ crawl: "c_81f2", links_to_status: "4xx" }
agent › 1,742 pages crawled. 23 link to a 404, all through the old /v1/ API reference. Want the list as a CSV?
Hook into every step of the crawl.
Four lifecycle hooks let you add your own checks, change how pages are fetched and rendered, or drop what you don’t need before it’s stored.
- 1
preAudit
Read and adjust the crawl configuration.
- 2
preCapture
Run just before a node is fetched. Skip it, or change the request.
- 3
render
Decide how each node is rendered: raw HTML or headless Chrome.
- 4
postCapture
Enrich, filter, score or export what was captured.
Send results wherever they’re going.
OpenCrawl doesn’t want to be your whole stack. It writes plain, documented formats, so the crawl slots in ahead of a search index, a warehouse, an eval set, or the crawler you already run.
- JSON
- One report per crawl, for scripts and CI.
- JSON Lines
- One record per page, streamed as it’s captured.
- SQLite
- Every run kept, queryable and comparable.
- MCP
- Live answers for the agents that need them.
Every URL accounted for.
Each URL is followed from discovery to the record you get out. Anything that drops out on the way, from a robots.txt rule to a duplicate to a page your own hook filtered, is counted and kept with its reason.
Where pages drop out
79% ofdiscovered pages end up delivered
Discovered
12,480 100%
Allowed
11,640 93%
Fetched
11,130 89%
Unique
10,210 82%
Delivered
9,870 79%
Hover a band to read it.
Raw HTML when it’s enough. A browser when it isn’t.
Rendering every page is slow and expensive. OpenCrawl fetches raw HTML by default, renders with headless Chrome where you ask it to, and shows exactly what the scripts added.
Render diff
3 of 9 blocks missing · 1,240 words
Drag the divider to compare. Hover a dashed block to name it.
- Missing without JavaScript: Plan comparison table, rendered by /assets/pricing.js
- Missing without JavaScript: FAQ answers, rendered by /assets/faq.js
- Missing without JavaScript: Customer quote, rendered by /assets/carousel.js
Watch a crawl as it runs.
Discovery and fetching are reported as they happen, against the last run of the same crawl, so slowdowns and missing sections show up early.
Pages discovered
- Discovered
- Fetched
- Last crawl
12,480
in 60s +4.3% vs last crawl
- Discovered
- Fetched
- Last crawl
| Second | Discovered | Fetched | Last crawl |
|---|---|---|---|
| 0 | 392 | 282 | 374 |
| 10 | 1576 | 1208 | 1487 |
| 20 | 4916 | 3998 | 4734 |
| 30 | 9121 | 7844 | 8872 |
| 40 | 11341 | 10283 | 11115 |
| 50 | 11950 | 11392 | 11757 |
| 60 | 12480 | 12480 | 11962 |
Fast without being rude.
A crawler that agents can start on their own has to behave well on its own. The defaults assume the site isn’t yours.
robots.txt first
Read before any other request and matched per RFC 9309, including wildcards and the longest-match rule.
Throttled by default
A few requests in flight, spaced apart. Turn it up for sites you own; leave it alone for the rest.
Honest about who it is
A clear user agent with a link back, so site owners know who’s visiting and how to reach you.