OpenCrawl is a fast, open web crawler built to be driven by AI agents over MCP, and to slot into whatever crawl pipeline you already run.
Model how each site should be crawled, test what comes back, and hand the results to your team and your agents. Every step is open and yours to change.
Shape the crawl to the site, not the other way round. Set scope, depth, rendering and pace per section, save it as a model, and reuse it on every crawl of that site.
Write down what a crawl should return and run it like a test suite. A changed template or a broken selector fails the run before bad data reaches your pipeline.
Every crawl ends in a report people can read and an API agents can query. Ask over MCP, or stream results into your warehouse, search index or dashboard.
Agents can start a crawl, check on it, and query what it found, as tool calls, without a human in between.
Hook in before a crawl, before and after each page is captured, and in the renderer itself. Filter, enrich, or send pages anywhere.
JSON, JSON Lines and SQLite, with stable fields. Feed an index, a warehouse, an eval set or the next step in your pipeline.
Raw HTML by default, headless Chrome for the pages that need it, and a diff showing what scripts add.
robots.txt (RFC 9309) is read before anything else, requests are throttled, and the crawler says who it is.
Crawls are stored and comparable, so you, or your agent, can ask what changed since last week.
From the CLI, from code, or by asking your agent. OpenCrawl reads robots.txt and sitemaps, then starts walking.
$ opencrawl crawl https://example.com robots.txt · 1,812 URLs queuedAdd the checks, filters and transforms your workflow needs, at the step where they belong.
postCapture(page) → extract /pricing · 3 plans extractedStream pages out as they’re captured, or let your agents query the finished crawl over MCP.
→ query_pages { status: "4xx" } 23 pages, 19 under /careers/The crawl engine is benchmarked across Go, Rust and TypeScript on every change, against a synthetic site and the live web.
HTML crawl vs JS rendering
2,587×
faster as raw HTML 37,663 vs 14.6 pages a second
| Time | HTML crawl | JS rendering |
|---|---|---|
| 1ms | 0 | 0 |
| 3ms | 0 | 0 |
| 10ms | 16 | 0 |
| 32ms | 313 | 0 |
| 103ms | 3185 | 0 |
| 327ms | 10000 | 0 |
| 1s | 10000 | 1 |
| 3.3s | 10000 | 33 |
| 10.6s | 10000 | 143 |
| 33.6s | 10000 | 484 |
| 60s | 10000 | 790 |