Truth-Recall: The Metric That Caps RAG Recall

By the fastCRW team · Benchmark figures verified 2026-05-18 against the run of record (diagnose_3way.py, 2026-05-08) · Verify independently before quoting internally.

Disclosure: we build fastCRW. This is a vendor-authored explainer, so weight it accordingly — but the numbers below are from a public Firecrawl dataset, the harness is named, and we publish the row where Firecrawl beats us. A benchmark that hides its own losses is marketing, not measurement.

Truth-recall is the web extraction metric that caps RAG recall

Most RAG teams tune the wrong layer. They swap embedding models, add a re-ranker, fiddle with chunk size — and never check whether the content they indexed actually contains the answer. The metric that decides that is truth-recall: of a set of pages with known ground-truth content, how much of that content did the extractor actually recover into clean markdown? If the extractor dropped the paragraph that holds the answer, no embedding model on earth retrieves it. Truth-recall is upstream of everything else in your pipeline, and it sets a hard ceiling on retrieval recall that you cannot raise by tuning the vector store.

This is an accuracy explainer, not a build tutorial. If you want the step-by-step LlamaIndex wiring, read the existing scrape-to-RAG pipeline with LlamaIndex guide. Here we define the metric, walk the three-way comparison with full provenance, and show how to read a recall benchmark without being misled.

What truth-recall measures (and what it does not)

Truth-recall vs scrape-success: two different denominators

These two numbers get conflated constantly, and they have different denominators, so conflating them is a category error. Scrape-success asks: of the URLs requested, how many returned a 2xx with some content? It is measured over the full request set. Truth-recall asks: of the pages where we know what the correct content is, how much of that content did we actually capture? It is measured only over the labeled subset that carries ground truth. A scraper can score high on the first and low on the second — it returned a page, just not the right content.

Why a high success rate can still mean missed content

Picture a docs page behind a JavaScript tab widget. A scraper fetches the shell, returns HTTP 200, and logs a success. But the answer lived in a tab that never rendered, so the markdown is missing the one section your user will ask about. Scrape-success: 100%. Truth-recall on that page: a fraction. For RAG, the second number is the one that determines whether your agent can answer — the first just tells you the request did not error.

The 819-labeled-URL ground-truth set behind the number

The recall figures here come from Firecrawl's own public scrape-content-dataset-v1, which contains 1,000 URLs. Of those, 819 carry labeled ground truth — a known-correct content target you can score an extraction against. That 819 is the accuracy denominator. We phrase recall as "of 819 labeled URLs" precisely so the number is not silently inflated by dividing against the full 1,000 or the 3,000 total requests. Same dataset for all three tools, same harness, same run.

The three-way comparison, with full provenance

fastCRW 63.74%, Crawl4AI 59.95%, Firecrawl 56.04%

On the 819 labeled URLs, fastCRW recovered the most ground-truth content of the three tools tested (diagnose_3way.py, single run, 3,000 requests, 2026-05-08):

fastCRW leads recall by +3.79 percentage points over Crawl4AI and +7.70 over Firecrawl. That is the headline for a RAG ingestion decision: more of the known-correct content survived into the index.

Pairing recall with scrape-success and 0 errors

All three tools threw 0 errors across 3,000 requests. "0 errors" sounds impressive in isolation, but it only means nothing crashed — it says nothing about whether the content was complete. fastCRW's standalone scrape-success was 91.8% of reachable URLs, 0 thrown errors — the relevant paired figure to anchor on alongside truth-recall. This separation of "returned a page" from "returned the right content" is the core point: a scraper can return a page with missing content and still count as a "success" on a naive metric.

Why we never cite recall as "of 1,000" or "of 3,000"

If you have seen larger-sounding recall denominators elsewhere, be skeptical. Truth-recall can only be computed where a labeled ground truth exists, and that is 819 URLs, not the 1,000 in the dataset and certainly not the 3,000 total requests (1,000 URLs × 3 tools). Dividing the 522 correctly-recovered fastCRW pages by 3,000 would produce a meaningless 17.4% that compares apples to nothing. Always check the denominator before you trust a recall number — yours or anyone's.

Why ingestion accuracy caps RAG retrieval recall

You cannot retrieve content that never made it into the index

This is the whole argument in one sentence. A vector store retrieves over the chunks you put in it. If extraction dropped a section, there is no chunk, no embedding, and no possible query that surfaces it. Your retrieval recall is bounded above by your extraction recall. A re-ranker can reorder what you have; it cannot conjure what you never captured.

Extraction recall is upstream of embeddings and re-ranking

Order matters. Extraction happens first, embeddings second, retrieval third, re-ranking fourth. Every later stage operates on the output of extraction. So spending your first optimization week on embedding-model A/B tests while ingesting at 56% recall is optimizing stage two while stage one quietly caps the whole pipeline. Fix the source of truth before you tune the math on top of it.

How a few missed sections become wrong answers

The failure is not loud. A missing section does not throw — it just is not there. The model retrieves the closest available chunk, which is plausibly worded and confidently wrong, and your evaluation set may not even cover that question. Multiply a handful of missed sections across thousands of ingested pages and you get a RAG system that is right most of the time and unaccountably, untraceably wrong on the long tail. Higher extraction recall is the cheapest way to shrink that tail. For why clean markdown specifically improves chunk quality, see LLM-ready markdown extraction.

Reading a truth-recall benchmark without being misled

Latency: fastest median and lowest p90 in fast mode

fastCRW leads on both headline latency metrics:

The p50 of 1,914 ms beats Firecrawl's 2,305 ms and is effectively tied with Crawl4AI. In fast mode, the p90 of 4,348 ms is the lowest of the three tools — significantly under Crawl4AI's 4,754 ms and Firecrawl's 6,937 ms. The chrome-stealth fallback that recovers the hard pages the other tools miss is the same mechanism behind the higher truth-recall; enabling full recall mode extends the tail for those harder pages. We publish the full latency split so you can decide what fits your pipeline.

Single run, 3,000 requests — what a point-in-time measurement proves

This is one run of 3,000 requests on one dataset on one day. It is a strong, reproducible signal that fastCRW recovered more labeled content than the other two tools under identical conditions. It is not a universal accuracy guarantee for every site, every region, or every point in time. The web changes, anti-bot postures change, and your URL mix is not this dataset. Treat the number as evidence to run your own ingestion-source comparison, not as a permanent law.

When success rate matters more than recall, and when it does not

Recall is not always the metric you optimize. If you are building uptime monitoring or link-checking, scrape-success is what you care about — you need to know a page is reachable, not capture every paragraph. For RAG ingestion, recall dominates: a reachable page with half its content is worse than useless because it indexes confidently and answers wrongly. Pick the metric that matches the job. For the full benchmark methodology and raw splits, see the fastCRW benchmark write-up and the public /benchmarks page.

Applying the metric in your pipeline

Choose the ingestion source before tuning embeddings

Sequence your optimization to match the dependency chain. Run a small labeled set of your own representative URLs through your candidate extractors, score truth-recall, and pick the highest scorer as your ingestion source first. Only then move on to embedding models and re-rankers. You will get more answer-quality lift from a 7-point recall gain at ingestion than from most embedding swaps, because the recall gain raises the ceiling rather than rearranging what is under it.

Feed the highest-recall markdown into your existing reader

Because the metric lives at ingestion, you do not have to rebuild your pipeline to benefit — you swap the source feeding it. fastCRW returns clean markdown over a Firecrawl-compatible REST surface, so in most pipelines pointing your reader at it is a base-URL change, not a rewrite. The downstream chunker, embedder, and retriever stay exactly as they are; only the quality of what enters them improves. To wire the LlamaIndex path end to end, the LlamaIndex build tutorial covers the reader and index steps.

Benchmark scope and engine details

One dataset, one run — not a universal guarantee

To restate it plainly so nobody over-reads the headline: 63.74% is fastCRW's recall on this dataset, on this run. It is the highest of the three tools tested under identical conditions, and it is a public, reproducible figure — but it does not promise 63.74% on your corpus. Run the comparison on your own labeled URLs before you standardize on any source, including ours.

Extraction, rendering, and session model at a glance

fastCRW's LLM-backed JSON extraction (formats: ["json"]) accepts up to 50 URLs per /v1/extract request. Screenshots are supported too: a formats: ["screenshot"] request returns a CDP-captured base64 PNG. Requests are stateless per call — there is no persistent session, so any cross-page state lives in your pipeline, not the engine — and LLM extraction runs on fastCRW's managed LLM, on paid plans only. None of this changes the recall story above; it just shapes how you build around it.

Sources

• Dataset: Firecrawl's public scrape-content-dataset-v1 (1,000 URLs, 819 labeled) — github.com/mendableai/firecrawl
• fastCRW open-core engine and API surface: github.com/us/crw · fastcrw.com

Related: Scrape-to-RAG pipeline with LlamaIndex · The fastCRW benchmark · LLM-ready markdown extraction