Skip to main content
Engineering

Data Quality & Truth-Recall for RAG Accuracy

RAG accuracy starts at extraction. Learn what truth-recall measures, how to benchmark scrape quality, and why clean source data caps your RAG answer quality.

fastcrw
By RecepNovember 21, 20268 min read

By the fastCRW team · Benchmark figures verified against the 3-way run of record (diagnose_3way.py, 2026-05-08) · Verify independently before quoting.

Data quality for RAG: accuracy is capped by what you extracted

Data quality for RAG web scraping accuracy is the first lever most teams ignore. When a retrieval-augmented-generation system answers wrong, the instinct is to tune the retriever: bigger embeddings, smarter re-ranking, larger top-k, a better chunking strategy. Those help, but they all operate on data that was already extracted. If the source page was scraped with the nav, the cookie banner, half the article body missing, or boilerplate masquerading as content, no retriever can recover the facts that never made it into the index. Retrieval can only surface what extraction captured.

This is why the most reliable predictor of RAG answer quality is not the vector database or the model — it is the accuracy of the scrape that produced your corpus in the first place. The cleaner and more complete the extracted text, the higher the ceiling on every downstream stage.

Why retrieval can't fix bad source data

Embedding-based retrieval matches a query against vectors derived from your extracted text. If a key sentence was dropped during extraction, there is no vector that represents it, so it can never be retrieved — the failure is silent. Worse, boilerplate that survived extraction (footers, related-article lists, legal disclaimers) gets embedded too, competing for top-k slots and diluting relevance. Garbage in the corpus is not neutral; it actively crowds out the signal. Cleaning at retrieval time is too late.

Truth-recall vs scrape-success: two different numbers

Teams routinely conflate two metrics that measure very different things. Scrape-success asks: did the request return a non-error response at all? Truth-recall asks: of the facts known to be on the page, how many did the extractor actually capture? A scraper can post a high success rate while returning thin, lossy content — every request "succeeds," but the body is half-empty. For RAG, truth-recall is the number that bounds answer quality, because it measures whether the facts your model needs are present in the corpus at all.

What truth-recall actually measures

Truth-recall is an accuracy metric, not a uptime metric. It requires a labeled ground-truth set: a collection of URLs where a human (or a trusted reference) has marked which facts or content blocks should appear in a correct extraction. You then run your scraper and measure what fraction of those labeled facts the extracted output contains. The result is a percentage of the labeled set — and the denominator matters enormously.

On the 3-way run of record, fastCRW posted the highest truth-recall of the three tools tested — 63.74% (522 of 819 labeled URLs), +3.79 pp over Crawl4AI and +7.70 pp over Firecrawl (diagnose_3way.py, on Firecrawl's own public 1,000-URL dataset, 2026-05-08). That is the number to weigh if your corpus quality is your bottleneck: it is the share of known-good content the extractor actually recovered.

Labeled ground truth as the denominator

The dataset above contains 1,000 URLs, but only 819 carry labeled ground truth. Truth-recall is computed against those 819 — not against the full 1,000, and not against the 3,000 total requests across the three tools. Using a larger denominator would silently inflate or deflate the figure depending on which way you rounded. The labeled subset is the only honest denominator because it is the only set where "did we capture the right facts?" has a defined answer.

Why you must report the denominator honestly

A recall percentage with no denominator is meaningless — "90% recall" of ten cherry-picked URLs tells you nothing about a million-page crawl. When you benchmark for your own pipeline, always state the size and provenance of the labeled set, the harness that produced the number, and the date. We phrase ours as "63.74% of 819 labeled URLs (diagnose_3way.py, 2026-05-08)" precisely so the claim is auditable. Demand the same rigor from any vendor's recall claim before you trust it.

Benchmarking scrape quality yourself

You do not have to take anyone's published recall figure on faith, and for your specific URL mix you shouldn't. The pages your RAG system ingests — your docs, your competitors' blogs, the knowledge bases in your domain — may behave very differently from a public benchmark dataset. The only number that matters for your pipeline is the one measured on your traffic.

Building a labeled evaluation set

Start small and honest. Pick 50–200 representative URLs from your real corpus. For each, record the facts or content blocks a correct extraction must contain — a few key sentences, a price, a table, the full article body. This is your ground truth. Then run each candidate extractor over the set and score what fraction of labeled facts each output captured. Two hundred labeled URLs is enough to separate a 60%-recall extractor from an 85%-recall one with confidence; you do not need thousands to make a defensible choice.

Pairing success rate with error count

Report success and errors together — one without the other is misleading. On the same 3-way run, fastCRW returned 91.8% scrape-success of reachable URLs with 0 thrown errors across all 3,000 requests (diagnose_3way.py, 2026-05-08). "0 errors" on its own would overstate the result; paired with the ~92% success rate it tells the true story: every request completed cleanly, and every reachable URL returned usable content. When you benchmark your own set, always publish the pair. A tool that "never errors" but quietly returns empty bodies on 30% of pages is worse for RAG than one that fails loudly — at least the loud failure can be retried.

Reading latency alongside accuracy

Accuracy is the ceiling on RAG quality, but latency governs whether a live-context pipeline is usable at all. Reporting one without the other hides trade-offs that matter at scale — and an honest benchmark publishes the full distribution, not a single flattering average.

Why median and tail both matter

A single "average latency" number is the most common way benchmarks mislead. The median (p50) tells you the typical request; the tail (p90, p99) tells you what your worst pages cost. On the 3-way run, fastCRW's p50 was 1914 ms, beating Firecrawl's 2305 ms and effectively tied with Crawl4AI (diagnose_3way.py, 2026-05-08). The typical page is fast. But the tail is a different story, and we publish it plainly.

Fast mode p90: the lowest of the three

In fast mode, fastCRW's p90 is 4348 ms — the lowest of the three tools tested (Crawl4AI 4754 ms, Firecrawl 6937 ms; diagnose_3way.py, 2026-05-08). The fast path covers the clear majority of URLs and delivers both the best truth-recall and the best tail latency. For the rare, difficult pages where fastCRW engages chrome-stealth recovery to retrieve content the others miss entirely, latency rises — but so does accuracy: fastCRW is the only tool to recover 34 URLs that neither Crawl4AI nor Firecrawl reach, 70% more exclusive recoveries than the other two combined. Knowing which URLs fall on each path is the whole decision. See the full p50/p90 split on /benchmarks and the methodology writeup in how we benchmark fastCRW.

Improving data quality upstream

Once you accept that RAG accuracy is downstream of extraction accuracy, the highest-leverage fixes move to the top of the pipeline — before a single vector is computed.

Clean markdown and dedup

Extracting to clean markdown rather than raw HTML strips the boilerplate that pollutes embeddings: scripts, styles, nav, footers, and tracking markup never enter the corpus. Clean markdown also makes chunking deterministic — headings become natural chunk boundaries instead of arbitrary character offsets through tag soup. Deduplicate aggressively: the same syndicated article scraped from five domains should contribute one chunk, not five near-identical vectors competing in retrieval. For the extraction-substrate argument in depth, see converting websites to clean markdown for LLMs, and for how clean boundaries feed retrieval, the scrape-to-RAG pipeline with LlamaIndex.

Choosing an extractor on accuracy, not vibes

The takeaway is simple: choose your extractor the way you would choose your model — on a measured number against a labeled set, not on a landing-page adjective. Build the small evaluation set, score the candidates on truth-recall paired with success and error counts, and read latency as a distribution. fastCRW leads on truth-recall (63.74% of 819 labeled URLs), median latency, and fast-mode p90 (4348 ms, lowest of the three); whether the details of your workload favour fast mode or chrome-stealth recovery depends on your URL mix. Either way, the discipline — labeled denominators, paired metrics, full distributions — is what separates a corpus you can trust from one you merely hope is clean. For the end-to-end ingest pattern, see building a RAG pipeline with CRW.

Sources

  • fastCRW 3-way scrape benchmark (truth-recall, scrape-success, p50/p90/p99): fastcrw.com/benchmarks — diagnose_3way.py, Firecrawl's public 1,000-URL dataset (819 labeled), 2026-05-08
  • Firecrawl public scrape-content dataset: firecrawl.dev (verified 2026-05-08)
  • fastCRW repo: github.com/fastcrw/crw (AGPL-3.0)

Related: How we benchmark fastCRW · Convert websites to clean markdown for LLMs · Scrape-to-RAG with LlamaIndex and CRW · RAG pipeline with CRW

FAQ

Frequently asked questions

What is truth-recall in web scraping?
Truth-recall measures, of the facts known to be on a page, how many an extractor actually captured. It requires a labeled ground-truth set and is reported as a percentage of that labeled set. On the 3-way run of record, fastCRW posted 63.74% truth-recall of 819 labeled URLs (diagnose_3way.py, 2026-05-08) — the highest of the three tools tested. It differs from scrape-success, which only asks whether a request returned without an error.
How does extraction quality affect RAG accuracy?
RAG answer quality is capped by what extraction captured. A retriever can only surface content that made it into the index; facts dropped during extraction are silently unretrievable, and surviving boilerplate dilutes relevance by competing for top-k slots. That is why source-extraction accuracy — not the vector database or model — is usually the strongest predictor of whether a RAG system answers correctly.
What is the difference between truth-recall and scrape-success?
Scrape-success asks whether a request returned a non-error response at all; truth-recall asks how many of the page's known facts the extractor actually captured. A scraper can show high success while returning thin, lossy content. Report them together: fastCRW returned 91.8% scrape-success of reachable URLs with 0 thrown errors across 3,000 requests, and separately 63.74% truth-recall of 819 labeled URLs (diagnose_3way.py, 2026-05-08).
How do I benchmark scrape quality for my RAG pipeline?
Pick 50–200 representative URLs from your real corpus, label the facts each correct extraction must contain, then run candidate extractors and score the fraction of labeled facts captured. Always state the size and provenance of the labeled set, pair success rate with error count, and read latency as a p50/p90/p99 distribution rather than a single average. Two hundred labeled URLs is enough to separate a 60%-recall extractor from an 85%-recall one with confidence.
What is fastCRW's p90 latency in fast mode?
In fast mode, fastCRW's p90 is 4348 ms — the lowest of the three tools tested (Crawl4AI 4754 ms, Firecrawl 6937 ms; diagnose_3way.py, 2026-05-08). The fast path covers the clear majority of URLs and delivers both the best truth-recall and the best tail latency. For the minority of difficult pages where chrome-stealth recovery is used, latency rises — but fastCRW recovers 34 URLs that neither competitor reaches, 70% more exclusive recoveries than the other two combined.

Get Started

Try fastCRW free

Run a live request in the playground — no signup required. Or grab a free API key with 1000 free credits, no credit card.

Continue exploring

More engineering posts

View category archive