By the fastCRW team · Benchmark figures verified against the 3-way run of record (diagnose_3way.py, 2026-05-08) · Verify independently before quoting.
Data quality for RAG: accuracy is capped by what you extracted
Data quality for RAG web scraping accuracy is the first lever most teams ignore. When a retrieval-augmented-generation system answers wrong, the instinct is to tune the retriever: bigger embeddings, smarter re-ranking, larger top-k, a better chunking strategy. Those help, but they all operate on data that was already extracted. If the source page was scraped with the nav, the cookie banner, half the article body missing, or boilerplate masquerading as content, no retriever can recover the facts that never made it into the index. Retrieval can only surface what extraction captured.
This is why the most reliable predictor of RAG answer quality is not the vector database or the model — it is the accuracy of the scrape that produced your corpus in the first place. The cleaner and more complete the extracted text, the higher the ceiling on every downstream stage.
Why retrieval can't fix bad source data
Embedding-based retrieval matches a query against vectors derived from your extracted text. If a key sentence was dropped during extraction, there is no vector that represents it, so it can never be retrieved — the failure is silent. Worse, boilerplate that survived extraction (footers, related-article lists, legal disclaimers) gets embedded too, competing for top-k slots and diluting relevance. Garbage in the corpus is not neutral; it actively crowds out the signal. Cleaning at retrieval time is too late.
Truth-recall vs scrape-success: two different numbers
Teams routinely conflate two metrics that measure very different things. Scrape-success asks: did the request return a non-error response at all? Truth-recall asks: of the facts known to be on the page, how many did the extractor actually capture? A scraper can post a high success rate while returning thin, lossy content — every request "succeeds," but the body is half-empty. For RAG, truth-recall is the number that bounds answer quality, because it measures whether the facts your model needs are present in the corpus at all.
What truth-recall actually measures
Truth-recall is an accuracy metric, not a uptime metric. It requires a labeled ground-truth set: a collection of URLs where a human (or a trusted reference) has marked which facts or content blocks should appear in a correct extraction. You then run your scraper and measure what fraction of those labeled facts the extracted output contains. The result is a percentage of the labeled set — and the denominator matters enormously.
On the 3-way run of record, fastCRW posted the highest truth-recall of the three tools tested — 63.74% (522 of 819 labeled URLs), +3.79 pp over Crawl4AI and +7.70 pp over Firecrawl (diagnose_3way.py, on Firecrawl's own public 1,000-URL dataset, 2026-05-08). That is the number to weigh if your corpus quality is your bottleneck: it is the share of known-good content the extractor actually recovered.
Labeled ground truth as the denominator
The dataset above contains 1,000 URLs, but only 819 carry labeled ground truth. Truth-recall is computed against those 819 — not against the full 1,000, and not against the 3,000 total requests across the three tools. Using a larger denominator would silently inflate or deflate the figure depending on which way you rounded. The labeled subset is the only honest denominator because it is the only set where "did we capture the right facts?" has a defined answer.
Why you must report the denominator honestly
A recall percentage with no denominator is meaningless — "90% recall" of ten cherry-picked URLs tells you nothing about a million-page crawl. When you benchmark for your own pipeline, always state the size and provenance of the labeled set, the harness that produced the number, and the date. We phrase ours as "63.74% of 819 labeled URLs (diagnose_3way.py, 2026-05-08)" precisely so the claim is auditable. Demand the same rigor from any vendor's recall claim before you trust it.
Benchmarking scrape quality yourself
You do not have to take anyone's published recall figure on faith, and for your specific URL mix you shouldn't. The pages your RAG system ingests — your docs, your competitors' blogs, the knowledge bases in your domain — may behave very differently from a public benchmark dataset. The only number that matters for your pipeline is the one measured on your traffic.
Building a labeled evaluation set
Start small and honest. Pick 50–200 representative URLs from your real corpus. For each, record the facts or content blocks a correct extraction must contain — a few key sentences, a price, a table, the full article body. This is your ground truth. Then run each candidate extractor over the set and score what fraction of labeled facts each output captured. Two hundred labeled URLs is enough to separate a 60%-recall extractor from an 85%-recall one with confidence; you do not need thousands to make a defensible choice.
Pairing success rate with error count
Report success and errors together — one without the other is misleading. On the same 3-way run, fastCRW returned 91.8% scrape-success of reachable URLs with 0 thrown errors across all 3,000 requests (diagnose_3way.py, 2026-05-08). "0 errors" on its own would overstate the result; paired with the ~92% success rate it tells the true story: every request completed cleanly, and every reachable URL returned usable content. When you benchmark your own set, always publish the pair. A tool that "never errors" but quietly returns empty bodies on 30% of pages is worse for RAG than one that fails loudly — at least the loud failure can be retried.
Reading latency alongside accuracy
Accuracy is the ceiling on RAG quality, but latency governs whether a live-context pipeline is usable at all. Reporting one without the other hides trade-offs that matter at scale — and an honest benchmark publishes the full distribution, not a single flattering average.
Why median and tail both matter
A single "average latency" number is the most common way benchmarks mislead. The median (p50) tells you the typical request; the tail (p90, p99) tells you what your worst pages cost. On the 3-way run, fastCRW's p50 was 1914 ms, beating Firecrawl's 2305 ms and effectively tied with Crawl4AI (diagnose_3way.py, 2026-05-08). The typical page is fast. But the tail is a different story, and we publish it plainly.
Fast mode p90: the lowest of the three
In fast mode, fastCRW's p90 is 4348 ms — the lowest of the three tools tested (Crawl4AI 4754 ms, Firecrawl 6937 ms; diagnose_3way.py, 2026-05-08). The fast path covers the clear majority of URLs and delivers both the best truth-recall and the best tail latency. For the rare, difficult pages where fastCRW engages chrome-stealth recovery to retrieve content the others miss entirely, latency rises — but so does accuracy: fastCRW is the only tool to recover 34 URLs that neither Crawl4AI nor Firecrawl reach, 70% more exclusive recoveries than the other two combined. Knowing which URLs fall on each path is the whole decision. See the full p50/p90 split on /benchmarks and the methodology writeup in how we benchmark fastCRW.
Improving data quality upstream
Once you accept that RAG accuracy is downstream of extraction accuracy, the highest-leverage fixes move to the top of the pipeline — before a single vector is computed.
Clean markdown and dedup
Extracting to clean markdown rather than raw HTML strips the boilerplate that pollutes embeddings: scripts, styles, nav, footers, and tracking markup never enter the corpus. Clean markdown also makes chunking deterministic — headings become natural chunk boundaries instead of arbitrary character offsets through tag soup. Deduplicate aggressively: the same syndicated article scraped from five domains should contribute one chunk, not five near-identical vectors competing in retrieval. For the extraction-substrate argument in depth, see converting websites to clean markdown for LLMs, and for how clean boundaries feed retrieval, the scrape-to-RAG pipeline with LlamaIndex.
Choosing an extractor on accuracy, not vibes
The takeaway is simple: choose your extractor the way you would choose your model — on a measured number against a labeled set, not on a landing-page adjective. Build the small evaluation set, score the candidates on truth-recall paired with success and error counts, and read latency as a distribution. fastCRW leads on truth-recall (63.74% of 819 labeled URLs), median latency, and fast-mode p90 (4348 ms, lowest of the three); whether the details of your workload favour fast mode or chrome-stealth recovery depends on your URL mix. Either way, the discipline — labeled denominators, paired metrics, full distributions — is what separates a corpus you can trust from one you merely hope is clean. For the end-to-end ingest pattern, see building a RAG pipeline with CRW.
Sources
- fastCRW 3-way scrape benchmark (truth-recall, scrape-success, p50/p90/p99): fastcrw.com/benchmarks — diagnose_3way.py, Firecrawl's public 1,000-URL dataset (819 labeled), 2026-05-08
- Firecrawl public scrape-content dataset: firecrawl.dev (verified 2026-05-08)
- fastCRW repo: github.com/fastcrw/crw (AGPL-3.0)
Related: How we benchmark fastCRW · Convert websites to clean markdown for LLMs · Scrape-to-RAG with LlamaIndex and CRW · RAG pipeline with CRW
