Rust Web Scraper for AI Agents: Feeding Fresh Web Content into Agent Context

Here is the recurring problem in one sentence: an agent is only as current as the last thing you put in its context, and the web is where "current" lives.

If you are building agents, you have already hit this. The model knows a lot, but it does not know what changed this morning. So you reach for a scraper, wire it into a tool call, and discover that the scraper is now the heaviest, flakiest, most operationally annoying part of your stack. It pulls in a headless Chrome, a queue, a couple of Node services, and a few gigabytes of container. You wanted fresh text for a RAG pipeline. You got a distributed system.

We built fastCRW as the opposite of that. It is a web scraper written in Rust, shipped as a single binary, that speaks an MCP interface so an agent can call it directly. This post is about using it for exactly one job: getting clean, fresh web content into an agent's context, whether that context is a RAG index or a live tool call mid-conversation. We will show the numbers we can stand behind, the MCP wiring, and the places it still falls short.

Why a scraper is an agent problem, not a data-engineering problem

Classic scraping is a batch job. You schedule it, it runs overnight, it fills a table. Agent scraping is interactive. The model decides, at inference time, that it needs a page, and it needs the readable content of that page in a form it can reason over, not raw HTML with nav bars and cookie banners.

That changes what matters:

• Latency at the median matters more than throughput. One page, right now, in the loop of a conversation.
• Content quality (truth-recall) matters more than raw success. A 200 response that returns a cookie wall is worse than useless: it poisons the context with noise the model will happily cite.
• Footprint matters because you often want the scraper to live next to the agent, not in a separate cluster you have to operate.

Those three pressures are what shaped fastCRW, so let's take them in order.

Truth-recall: getting the right content, not just a 200

Success rate is the metric everyone quotes and the one that lies the most. A scraper can "succeed" on a page and hand your agent a paywall stub, a consent banner, or an SPA shell with no article in it.

So the metric we lead with is truth-recall: of the pages that have a known, labeled correct answer, how many did the scraper actually recover the real content for?

On Firecrawl's public scrape-content-dataset-v1 (1,000 URLs, 819 labeled), measured with diagnose_3way.py in a single run of 3,000 requests on 2026-05-08:

• fastCRW: 63.74% truth-recall (522 of 819 labeled URLs)
• Crawl4AI: 59.95% (491 of 819)
• Firecrawl: 56.04% (459 of 819)

fastCRW recovered the most real content of the three, by +3.79 percentage points over Crawl4AI and +7.70 over Firecrawl. It is worth saying plainly that this is Firecrawl's own benchmark dataset, so this is not a home-field number.

Now the honest pairing, because "63.74% recall" without the raw scrape rate would be cherry-picking. Of the full 1,000 URLs, scrape-success was:

• fastCRW: 87.7% (877 of 1,000)
• Crawl4AI: 83.5% (835)
• Firecrawl: 89.7% (897, the highest of the three)

Firecrawl fetched more pages successfully. fastCRW extracted the right content from more pages. For an agent, the second number is the one that keeps garbage out of your context window. And across all 3,000 requests in that run, all three tools threw 0 errors, so the difference is genuinely about extraction quality, not crashes.

The reason we split these two numbers apart, always, is that "0 errors" on its own overstates the story. Zero crashes and 87.7% real-content recovery is the honest sentence.

Latency: fast at the median, slow in the tail (and why)

For an in-the-loop agent call, the median is what your users feel. In the same 3-way run:

• p50 latency: fastCRW 1914 ms (the fastest of the three, though effectively tied with Crawl4AI's 1916 ms, 2 ms apart). Firecrawl was 2305 ms.

So on a typical page, fastCRW is at the front. Here is the part we refuse to bury:

• p90 latency: fastCRW 14157 ms (the worst of the three). Crawl4AI 4754 ms (best), Firecrawl 6937 ms.
• p99 latency: fastCRW 15012 ms. Crawl4AI 13749 ms, Firecrawl 21107 ms.

That p90 is ugly and we are not going to pretend otherwise. It is also causal: the tail is the price of the recall win at the top of this post. When a lightweight fetch comes back thin, fastCRW falls back to a stealth Chrome render to recover the content that the other tools return an empty-but-successful 200 for. That fallback is slow. It is also why the 522-vs-459 recall gap exists.

The tradeoff, stated bluntly: if you need a hard latency ceiling on every single call, that tail will hurt you, and Crawl4AI's p90 is genuinely better. If you care more about not feeding your agent hollow pages, and you can tolerate a slow long-tail on the hard ones, fastCRW is the better shape. We publish the full p50/p90/p99 split every time rather than a single average, because a single average would hide exactly the number you most need to plan around.

Search: the fast path for "just find me sources"

A lot of agent web use is not "scrape this URL," it is "find me the current sources on X." fastCRW has a search endpoint tuned for that, and it is measured separately because it exercises search, not scrape.

Over a 100-query benchmark across 10 categories (benchmarks/triple-bench.ts, a single point-in-time run):

• fastCRW search averaged 880 ms; 73 of 100 latency wins. Firecrawl averaged 954 ms (25 wins), Tavily 2,000 ms (2 wins).
• All three returned a 100% success rate.

For the agent pattern of "search, pick the top few results, then scrape them," a sub-second median search leg keeps the whole tool-use turn snappy.

The footprint argument: put the scraper next to the agent

This is where being a Rust single binary stops being a bragging right and starts being an operational decision. These are structural facts from the README, not benchmark claims:

• fastCRW ships as a single ~8 MB binary. Firecrawl's stack is ~2 to 3 GB total.
• fastCRW runs in 1 container (plus an optional sidecar). Firecrawl runs 5.
• The how-we-built writeup cites ~50 MB RAM idle on a $5 VPS, versus a Crawl4AI/Playwright baseline cited as 300 MB+ idle and ~200 to 300 MB per worker.

Why this matters for agents specifically: an 8 MB binary that idles around 50 MB of RAM is something you can colocate with your agent, drop into the same pod, ship inside a Docker image without bloating it, or run on the cheapest VPS you have. You are not standing up a scraping cluster to answer one question per conversation.

The default Docker Compose ships lightpanda, a lightweight browser, and keeps Chrome as an opt-in variant (roughly a 500 MB image and ~1 GB resident when you enable it). So the heavy renderer is there when you need it for hard pages, and absent when you don't.

Wiring it into an agent: the MCP path

The reason this is a scraper for AI agents and not just a fast CLI is that it exposes an MCP interface, so a model can call it as a tool with no glue code from you. The core tools map to the core verbs:

• crw_scrape — one URL to clean content
• crw_search — query to ranked results
• crw_map — discover the URLs on a site
• crw_crawl + crw_check_crawl_status — walk a site, poll for completion

A typical RAG-ingest or in-context-refresh flow looks like this:

1. crw_search("<topic> 2026")        -> ranked result URLs
2. crw_scrape(url) for the top N      -> clean markdown per page
3. chunk + embed the markdown         -> your vector store
4. agent retrieves at query time      -> fresh content in context

Or, for a whole documentation site you want indexed:

1. crw_map("https://docs.example.com")     -> every URL on the site
2. crw_crawl(...) then crw_check_crawl_status(...)  -> poll to done
3. feed the returned markdown into your embedder

The output is readable content, which is the whole point: you are handing the model article text, not a DOM it has to mentally strip. And because the credit model is flat and predictable, you can reason about cost per ingest.

What it costs to run

Two ways to run it, and both are honest about price.

Self-host the engine. It is AGPL-3.0, so self-hosting is free: you pay only for your server. At that point a scrape costs $0 per 1,000 scrapes in software terms. For comparison, Firecrawl's hosted tiers run $0.83 to $5.33 per 1,000 scrapes (marketing/competitor-prices.lock.md, verified 2026-05-18). If you are ingesting a lot of pages into a RAG store, self-hosting an 8 MB binary is a very different cost curve than a metered hosted API.

Use the hosted API if you would rather not run anything. The credit costs are flat and easy to budget against:

• scrape (any renderer, including Chrome): 1 credit
• crawl: 1 per page · search: 1 per query · map: 1 · browse (an MCP session): 1 per session
• extract, or any request asking for formats: ["json"]: 5 credits

The free tier gives you 500 one-time lifetime credits to try it (not a monthly meter). For live plan pricing, see the pricing page rather than trusting a number hard-coded in a blog post.

One note on the LLM-backed features (search answers, summarize, structured extract): managed LLM usage is metered in credits based on usage, and it is available on paid plans only. The free tier has no LLM and will return a 402 if you ask for those features.

The honest gaps

If you are putting this in front of an agent, you need to know where it will let you down, so here they are plainly. This is the part that matters most for a production decision.

No PDF or DOCX parsing. fastCRW handles HTML pages. Point it at a PDF and you get empty or an error, not extracted text. If your agent's sources include PDFs (research, filings, manuals), route those to a dedicated parser like pdfplumber and only send HTML to fastCRW. PDF text extraction is on the roadmap but not this cycle; DOCX is further out.

Anti-bot is not best-in-class. fastCRW does the basics well: realistic user agents, header mimicry, cookie handling, delays. That is fine for the large majority of public content. But aggressively protected sites (Cloudflare Enterprise, PerimeterX, DataDome, Akamai Bot Manager) will block it more often than a specialized anti-bot service would. There is no magic evasion engine here, and CAPTCHA solving is not planned. The workaround is to bring your own residential proxy via the proxy field. Built-in proxy rotation is being evaluated. If your agent needs to read hostile sites at scale, plan for this.

SPA coverage via LightPanda is inconsistent. The default lightweight browser (LightPanda, an experimental Zig browser that implements a subset of the W3C APIs, not full Chromium) handles server-hydrated pages well, but a complex React/Vue/Angular app with heavy client-side routing and lazy loading can come back thin. A good detection heuristic: if you get under ~200 words back from a page you know is content-rich, you probably hit this. The fix is to force the chrome renderer for that request, or fall back to Playwright. This is the highest-priority core gap and it is under active development.

It is stateless, poll-based, and Firecrawl-shaped but not identical. Each request is independent: no persistent session, no WebSocket or SSE stream of crawl progress, so you poll crw_check_crawl_status rather than subscribe. There is no batched multi-URL extract (iterate or crawl instead), and a few response fields and error envelopes diverge slightly from Firecrawl's. If you are migrating from Firecrawl, the API is close enough to be a mostly-drop-in swap, but budget an afternoon to check the edges. robots.txt is respected by default, which you may or may not want.

Where this fits

If you want a hard latency SLA on every call, or you scrape hostile anti-bot sites, or your corpus is mostly PDFs, fastCRW is not your best pick today, and we would rather tell you that now than have you find out in production.

But if what you actually need is to feed fresh, clean, correctly-extracted web content into an agent, the case is strong: highest truth-recall of the three tools on Firecrawl's own dataset, the fastest median scrape and search, and a footprint small enough (a single ~8 MB binary) that you can run it right next to the agent instead of operating a scraping cluster to answer one question at a time. It is open (AGPL-3.0), it is local, and it speaks MCP so the model can just call it.

Start with the 500 free lifetime credits, wire up crw_search and crw_scrape as MCP tools, and see whether the recall difference shows up in your own retrieval quality. That is the number that will actually change your agent's answers.

Benchmark provenance: 3-way scrape figures are from Firecrawl's public scrape-content-dataset-v1 (1,000 URLs, 819 labeled), diagnose_3way.py, single run of 3,000 requests, 2026-05-08. Search figures are from a 100-query benchmark across 10 categories, benchmarks/triple-bench.ts, single point-in-time run. Footprint figures are structural facts from the README, not benchmark claims. Cost comparison from marketing/competitor-prices.lock.md, verified 2026-05-18.