Results from the 1,000-URL Firecrawl Dataset Benchmark

Summary

On Firecrawl's own public 1,000-URL dataset, fastCRW returned the most accurate content of the three tools tested. It recovered the labeled content on 63.74% of matchable URLs, ahead of Crawl4AI's 59.95% and Firecrawl's 56.04%. It did this at a p50 latency of 1914 ms — faster at the median than Firecrawl's 2305 ms — and in fast mode its p90 of 4348 ms is the lowest of the three.

This page is the canonical reference for those numbers. The benchmark is a 3-way run on Firecrawl's own published dataset, scored by an open harness, with a per-URL result of record anyone can audit.

What Was Measured

The dataset is scrape-content-dataset-v1 — 1,000 URLs published by Firecrawl for evaluating scrape quality. Of those, 819 carry labeled ground-truth content and form the accuracy denominator. All three tools — fastCRW, Crawl4AI, and Firecrawl — were run against the same 1,000 URLs, in the same conditions, with 3,000 total requests scored by the same harness.

The headline metric is truth-recall: did the tool actually return the page's real content? A scrape that returns 200 OK with an anti-bot interstitial "succeeds" but recovers nothing useful — so success rate alone is misleading. Truth-recall corrects for that.

Results

Source / provenance. Every number below is verbatim from the result of record, bench/server-runs/RESULT_3WAY_1000_FULL.md, a full 1,000-URL run dated 2026-05-08, scored by the open diagnose_3way.py harness. fastCRW does not measure competitors by hand here: all three tools run through the identical scoring pipeline.

Metric	fastCRW	Crawl4AI	Firecrawl
Truth-recall (recall mode, 522 of 819 labeled URLs)	63.74%	59.95%	56.04%
Thrown errors (of 3,000 requests)	0	0	0
p50 latency	1914 ms	1916 ms	2305 ms
p90 latency (fast mode)	4348 ms	4754 ms	6937 ms
p99 latency	15012 ms	13749 ms	21107 ms

Read the rows together, not in isolation:

• fastCRW leads on truth-recall by a clear margin — +3.79 points over Crawl4AI, +7.70 over Firecrawl. On accuracy, the metric that decides whether a RAG pipeline gets real content, fastCRW is first.
• fastCRW wins the median (p50) and in fast mode also wins the tail (p90). At p50 it is essentially tied with Crawl4AI and ~17% faster than Firecrawl. In fast mode its p90 of 4348 ms is the lowest of the three — the next section explains the fast-mode / recall-mode split.
• fastCRW achieves 91.8% scrape success of reachable URLs and 0 thrown errors across 3,000 requests. All three tools hit zero crashes. "Reachable" means the 955 URLs that at least one engine could fetch: 45 of the dataset's 1,000 URLs are dead links no tool reaches, and counting those against an engine measures the dataset, not the scraper. The raw per-URL results are in bench/server-runs/diag3w-1000-full.jsonl, so the denominator is reproducible from the published run.

Fast Mode vs Recall Mode

fastCRW exposes two operating points, both real and reproducible:

• Fast mode — no chrome-stealth fallback. p90: 4348 ms (lowest of the three). Truth-recall is lower because hard, blocked pages are not retried.
• Recall mode — full chrome-stealth fallback ladder. Truth-recall: 63.74% (highest of the three). Tail latency is higher because a minority of difficult pages trigger a full stealth-browser retry instead of being dropped.

The chrome-stealth retry that recovers URLs the others miss is the same mechanism that can extend tail latency on hard pages. The p90 in fast mode and the truth-recall lead in recall mode are two honest operating points of the same system — which one matters depends on your workload.

This is why the page reports the full p50/p90/p99 split instead of a single "average." An average would hide exactly the information a production team needs: most requests are fast (p50 1914 ms), the split between modes is disclosed, and the slow retry comes back with content instead of an error.

How to Read These Numbers

Use this benchmark as:

• a starting point for a Firecrawl-replacement evaluation,
• a source of concrete, sourced metrics to weigh against your own workload,
• evidence that fastCRW is the most accurate of the three on a neutral, Firecrawl-published dataset.

Do not use it as proof that fastCRW wins every category for every site. It does not. Product maturity in adjacent workflows differs. The honest claim is narrow and defensible: highest truth-recall of three tools, median latency ahead of Firecrawl, p90 lowest of three in fast mode — all fully disclosed.

What This Benchmark Does Not Prove

• It does not measure every site on the web — it measures Firecrawl's 1,000-URL sample.
• It does not measure bundled feature surface, dashboards, or support quality.
• It does not replace testing your own target sites — a benchmark is a prior, not a guarantee.

That is why this page should be read alongside the methodology page and the fastCRW vs Firecrawl comparison.

Reproduce It Yourself

The dataset is public and the harness is open source:

git clone https://github.com/us/crw
cd crw
# Firecrawl's scrape-content-dataset-v1 + diagnose_3way.py harness
python bench/diagnose_3way.py

Every per-URL outcome is recorded in RESULT_3WAY_1000_FULL.md.

Next Steps

• See pricing — managed plans and free self-hosting for the workload shown here.
• fastCRW vs Firecrawl — full feature, pricing, and migration comparison.
• Benchmark methodology — how fastCRW frames internal vs external numbers and sources its claims.
• Search benchmark — the separate 100-query search-API comparison.