How fast can you get through the maze?

The OpenCrawl engine is benchmarked across three runtimes by its own harness. These are the published results, unedited.

Same crawler, three runtimes

The crawl engine is implemented in Go, Rust and TypeScript, and each one runs the same crawl back to back.

Same-domain, breadth-first

Each crawler starts from one URL, follows redirects, and stays on the starting domain until it runs out of links.

Two targets

A synthetic local site measures the crawler on its own. A live website measures it against the real network.

Synthetic site

A local benchmark server with no network latency, so the only bottleneck is the crawler itself: parsing, link extraction, deduplication and scheduling.

Date: 15 March 2026
Passes: 1
Target: Local synthetic site (localhost)

33,450 req/s

Rust

32,405 req/s

TypeScript

14,230 req/s

Each counter runs at its runtime’s measured speed (synthetic site, single pass).

Go: 33,450.07 req/s
Rust: 32,404.53 req/s
TypeScript: 14,230.47 req/s

Source: report_20260315_225421.md

Live website

A same-domain crawl of a live production website, averaged over 20 passes. Here the network sets the pace, and every runtime reaches the same ceiling.

Date: 26 April 2026
Passes: 20
Target: Live production website, same-domain crawl

286 req/s

TypeScript

284 req/s

Rust

278 req/s

Each counter runs at its runtime’s measured speed (live website, 20-pass average).

Go: 285.65 req/s
TypeScript: 283.8 req/s
Rust: 277.75 req/s