Here is the recurring problem in one sentence: an agent is only as current as the last thing you put in its context, and the web is where "current" lives.
If you are building agents, you have already hit this. The model knows a lot, but it does not know what changed this morning. So you reach for a scraper, wire it into a tool call, and discover that the scraper is now the heaviest, flakiest, most operationally annoying part of your stack. It pulls in a headless Chrome, a queue, a couple of Node services, and a few gigabytes of container. You wanted fresh text for a RAG pipeline. You got a distributed system.
We built fastCRW as the opposite of that. It is a web scraper written in Rust, shipped as a single binary, that speaks an MCP interface so an agent can call it directly. This post is about using it for exactly one job: getting clean, fresh web content into an agent's context, whether that context is a RAG index or a live tool call mid-conversation. We will show the numbers we can stand behind and the MCP wiring.
Why a scraper is an agent problem, not a data-engineering problem
Classic scraping is a batch job. You schedule it, it runs overnight, it fills a table. Agent scraping is interactive. The model decides, at inference time, that it needs a page, and it needs the readable content of that page in a form it can reason over, not raw HTML with nav bars and cookie banners.
That changes what matters:
- Latency at the median matters more than throughput. One page, right now, in the loop of a conversation.
- Content quality (truth-recall) matters more than raw success. A 200 response that returns a cookie wall is worse than useless: it poisons the context with noise the model will happily cite.
- Footprint matters because you often want the scraper to live next to the agent, not in a separate cluster you have to operate.
Those three pressures are what shaped fastCRW, so let's take them in order.
Truth-recall: getting the right content, not just a 200
Success rate is the metric everyone quotes and the one that lies the most. A scraper can "succeed" on a page and hand your agent a paywall stub, a consent banner, or an SPA shell with no article in it.
So the metric we lead with is truth-recall: of the pages that have a known, labeled correct answer, how many did the scraper actually recover the real content for?
On Firecrawl's public scrape-content-dataset-v1 (1,000 URLs, 819 labeled), measured with diagnose_3way.py in a single run of 3,000 requests on 2026-05-08:
- fastCRW: 63.74% truth-recall (522 of 819 labeled URLs)
- Crawl4AI: 59.95% (491 of 819)
- Firecrawl: 56.04% (459 of 819)
fastCRW recovered the most real content of the three, by +3.79 percentage points over Crawl4AI and +7.70 over Firecrawl. It is worth saying plainly that this is Firecrawl's own benchmark dataset, so this is not a home-field number.
Pairing recall with the raw scrape rate matters, because a recall number without it would be cherry-picking. Of the 955 reachable URLs in the set (45 of the 1,000 are dead links no tool can fetch), scrape-success was fastCRW 91.8% (877 of 955) and Crawl4AI 87.4% (835 of 955). Across all 3,000 requests in that run, all three tools threw 0 errors, so the recall gap is genuinely about extraction quality, not crashes.
For an agent, extracting the right content is the number that keeps garbage out of your context window: a 200 response with an empty SPA shell or a cookie wall is worse than a clean failure, because the model will happily cite it.
Latency: fast at the median, slow in the tail (and why)
For an in-the-loop agent call, the median is what your users feel. In the same 3-way run:
- p50 latency: fastCRW 1914 ms (the fastest of the three, though effectively tied with Crawl4AI's 1916 ms, 2 ms apart). Firecrawl was 2305 ms.
So on a typical page, fastCRW is at the front. When a lightweight fetch comes back thin, fastCRW auto-escalates through its renderer ladder (LightPanda, then Chrome over CDP, then a proxied Chrome tier) to recover real content instead of handing back an empty-but-successful 200. That escalation is what drives the truth-recall win above: fastCRW keeps trying until it has the actual page, not just a status code.
Search: the fast path for "just find me sources"
A lot of agent web use is not "scrape this URL," it is "find me the current sources on X." fastCRW has a search endpoint tuned for that, and it is measured separately because it exercises search, not scrape.
Over a 100-query benchmark across 10 categories (benchmarks/triple-bench.ts, a single point-in-time run):
- fastCRW search averaged 880 ms; 73 of 100 latency wins. Firecrawl averaged 954 ms (25 wins), Tavily 2,000 ms (2 wins).
- All three returned a 100% success rate.
For the agent pattern of "search, pick the top few results, then scrape them," a sub-second median search leg keeps the whole tool-use turn snappy.
The footprint argument: put the scraper next to the agent
This is where being a Rust single binary stops being a bragging right and starts being an operational decision. These are structural facts from the README, not benchmark claims:
- fastCRW ships as a single ~8 MB binary. Firecrawl's stack is ~2 to 3 GB total.
- fastCRW runs in 1 container (plus an optional sidecar). Firecrawl runs 5.
- The how-we-built writeup cites ~50 MB RAM idle on a $5 VPS, versus a Crawl4AI/Playwright baseline cited as 300 MB+ idle and ~200 to 300 MB per worker.
Why this matters for agents specifically: an 8 MB binary that idles around 50 MB of RAM is something you can colocate with your agent, drop into the same pod, ship inside a Docker image without bloating it, or run on the cheapest VPS you have. You are not standing up a scraping cluster to answer one question per conversation.
The default Docker Compose ships lightpanda, a lightweight browser, and keeps Chrome as an opt-in variant (roughly a 500 MB image and ~1 GB resident when you enable it). So the heavy renderer is there when you need it for hard pages, and absent when you don't.
Wiring it into an agent: the MCP path
The reason this is a scraper for AI agents and not just a fast CLI is that it exposes an MCP interface, so a model can call it as a tool with no glue code from you. The core tools map to the core verbs:
crw_scrape— one URL to clean contentcrw_search— query to ranked resultscrw_map— discover the URLs on a sitecrw_crawl+crw_check_crawl_status— walk a site, poll for completion
A typical RAG-ingest or in-context-refresh flow looks like this:
1. crw_search("<topic> 2026") -> ranked result URLs
2. crw_scrape(url) for the top N -> clean markdown per page
3. chunk + embed the markdown -> your vector store
4. agent retrieves at query time -> fresh content in context
Or, for a whole documentation site you want indexed:
1. crw_map("https://docs.example.com") -> every URL on the site
2. crw_crawl(...) then crw_check_crawl_status(...) -> poll to done
3. feed the returned markdown into your embedder
The output is readable content, which is the whole point: you are handing the model article text, not a DOM it has to mentally strip. And because the credit model is flat and predictable, you can reason about cost per ingest.
What it costs to run
Two ways to run it, and both are honest about price.
Self-host the engine. It is AGPL-3.0, so self-hosting is free: you pay only for your server. At that point a scrape costs $0 per 1,000 scrapes in software terms. For comparison, Firecrawl's hosted tiers run $0.83 to $5.33 per 1,000 scrapes (marketing/competitor-prices.lock.md, verified 2026-05-18). If you are ingesting a lot of pages into a RAG store, self-hosting an 8 MB binary is a very different cost curve than a metered hosted API.
Use the hosted API if you would rather not run anything. The credit costs are simple and easy to budget against:
scrape(any renderer, including Chrome): 1 creditcrawl: 1 per page ·search: 1 per query ·map: 1 ·browse(an MCP session): 1 per sessionextract, or any request asking forformats: ["json"]: 1 scrape credit plus the LLM cost for that page, billed as usage-metered LLM credits
The free tier gives you 1000 one-time lifetime credits to try it (not a monthly meter). For live plan pricing, see the pricing page rather than trusting a number hard-coded in a blog post.
One note on the LLM-backed features (search answers, summarize, structured extract): managed LLM usage is metered in credits based on usage, and it is available on paid plans only. The free tier has no LLM and will return a 402 if you ask for those features.
What's built in for hard pages
If you are putting this in front of an agent, here is what handles the cases that break naive scrapers.
PDF parsing is built in. PDF URLs auto-route to a dedicated parser and come back as clean markdown, so research papers, filings, and manuals feed the same pipeline as HTML pages with no separate tool to wire up.
Anti-bot is built in, not bolted on. fastCRW ships 12-signal block detection, user-agent rotation, stealth fingerprints, and proxy rotation through a residential-proxy egress tier in the open core, all served by every render tier. No separate evasion service to run.
SPA rendering auto-escalates. Pages get promoted through an escalation ladder, HTTP-only fetch, then LightPanda, then Chrome over CDP, then a proxied Chrome tier, so a heavy React/Vue/Angular app with client-side routing gets a real render without you having to pick a renderer up front. An optional Camoufox stealth renderer is available for the hardest cases.
It is stateless, poll-based, and Firecrawl-shaped. Each request is independent: no persistent session, no WebSocket or SSE stream of crawl progress, so you poll crw_check_crawl_status rather than subscribe. /v1/extract accepts up to 50 URLs in a single request for batched structured extraction. A few response fields and error envelopes diverge slightly from Firecrawl's, so if you are migrating, budget a short afternoon to check the edges. robots.txt is respected by default, which you may or may not want.
Where this fits
If what you need is to feed fresh, clean, correctly-extracted web content into an agent, the case is strong: highest truth-recall of the three tools on Firecrawl's own dataset, the fastest median scrape and search, and a footprint small enough (a single ~8 MB binary) that you can run it right next to the agent instead of operating a scraping cluster to answer one question at a time. It is open (AGPL-3.0), it is local, and it speaks MCP so the model can just call it.
Start with the 1000 free lifetime credits, wire up crw_search and crw_scrape as MCP tools, and see whether the recall difference shows up in your own retrieval quality. That is the number that will actually change your agent's answers.
Benchmark provenance: 3-way scrape figures are from Firecrawl's public scrape-content-dataset-v1 (1,000 URLs, 819 labeled), diagnose_3way.py, single run of 3,000 requests, 2026-05-08. Search figures are from a 100-query benchmark across 10 categories, benchmarks/triple-bench.ts, single point-in-time run. Footprint figures are structural facts from the README, not benchmark claims. Cost comparison from marketing/competitor-prices.lock.md, verified 2026-05-18.