Best Firecrawl Alternative for Self-Hosting

Here is the recurring problem in one sentence: self-hosting a web scraping API should not mean standing up a five-container cluster and feeding it a few gigabytes of RAM before you have scraped a single page.

If you have tried to self-host Firecrawl, you know the shape of it. Lovely engineering, genuinely — but the footprint is built for their hosted platform, not for a box you own. fastCRW is the other end of that tradeoff: a self-hosted web scraping API that ships as a single ~8 MB Rust binary, idles at roughly 50 MB of RAM on a $5 VPS, and speaks a drop-in-compatible API so your existing client code mostly keeps working.

Two things make it worth a look before you reach for a cluster:

1. It is more open and more local. The engine is AGPL-3.0. You run the binary, your data never leaves your box, and there is no per-scrape meter.
2. It is accurate where it counts. On Firecrawl's own public benchmark, fastCRW recovered more of the labeled content than either Firecrawl or Crawl4AI. Numbers and provenance below — we do not ask you to take that on faith.

This post covers how self-hosting actually works, what the footprint and accuracy numbers really say, and when you should self-host versus let someone else run it.

The footprint: what "self-hosted" usually costs, and what it costs here

These are structural facts from the README (§Structural footprint), not benchmark claims — but they are the whole reason this post exists.

For comparison, a Crawl4AI/Playwright-style setup is cited at 300 MB+ idle RAM and roughly 200–300 MB per worker (how-we-built-fastcrw). None of these are wrong tools — Playwright is excellent, and Crawl4AI is a pleasure inside a Python notebook. But if your goal is "a scraping API on a small box that I never have to babysit," an 8 MB binary that idles at ~50 MB is a different category of thing than 5 containers and a couple of gigabytes.

Worth knowing up front: the default binary uses LightPanda, an experimental Zig-based browser that implements a subset of the W3C APIs. It is not Chromium. That keeps the image tiny, and it handles server-rendered and server-hydrated pages well — see the configuration notes below for heavy client-side apps. There is an opt-in Chrome Compose variant (~500 MB image, ~1 GB resident) for when you need a full browser. You opt into that weight only when a page actually demands it.

How self-hosting works

The moving parts are deliberately few.

1. Run the binary. The default Compose file brings up one container running the binary with the LightPanda renderer. No queue service, no separate browser pool, no Redis-shaped dependency to keep alive.

2. Call a Firecrawl-shaped API. Your existing scrape/crawl/map/search calls port over with minor changes. A basic scrape:

curl -X POST http://localhost:3002/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "formats": ["markdown"]
  }'

3. Pick a renderer when you need to. The engine selects a renderer automatically, but you can force one. For a client-heavy page that comes back thin under LightPanda, force Chrome (this requires the opt-in Chrome variant):

curl -X POST http://localhost:3002/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://app.example.com/dashboard",
    "formats": ["markdown"],
    "renderer": "chrome"
  }'

4. There is no per-scrape bill. When you self-host the AGPL-3.0 engine, scrapes are free — you pay only for your server. The credit costs that apply on the managed service (scrape = 1 credit as of 2026-06-24, crawl = 1 per page, search = 1 per query, map = 1, an MCP browse session = 1, and any formats: ["json"]/extract request = 5, per the README §Renderer selection) are the managed metering model, not a toll you pay when you run the binary yourself.

That is the entire mental model: one process, a familiar API, renderer control when a page fights back, and no meter.

The accuracy story, on Firecrawl's own benchmark

This is the part that should decide whether an 8 MB binary is a serious tool or just a small one. We ran all three engines against Firecrawl's public scrape-content-dataset-v1 (1,000 URLs, 819 of them labeled), using diagnose_3way.py, in a single run of 3,000 requests on 2026-05-08. Every number in this section comes from that run.

Truth-recall, of the 819 labeled URLs — how much of the content that was supposed to be there did each engine actually recover:

• fastCRW: 63.74% (522) — highest of the three
• Crawl4AI: 59.95% (491)
• Firecrawl: 56.04% (459)

That is +3.79 pp over Crawl4AI and +7.70 pp over Firecrawl on Firecrawl's own dataset. Recall is the metric we optimize for, because a scrape that returns HTTP 200 with half the article missing is a silent failure — it looks like success and poisons whatever you feed it into.

Scrape-success, of reachable URLs — did a request come back with usable content at all:

• fastCRW: 91.8% (877 of 955 reachable) — 0 thrown errors
• Crawl4AI: 83.5% (835 of 1,000)

Recall asks "how complete," success asks "did anything come back" — fastCRW leads on both when measured against the reachable portion of the set. The 0 thrown errors across all 3,000 requests pairs with that 91.8%, since "zero errors" alone would overstate the case — a thin-but-successful response throws no error either.

The latency picture, in two modes

Scrape latency, same run (diagnose_3way.py, 2026-05-08):

At the median, fastCRW is the fastest of the three — effectively tied with Crawl4AI, 2 ms apart. The default mode chases hard pages with a Chrome-stealth fallback rather than giving up on them, which is a large part of how it wins truth-recall above. For workloads that want a tighter tail instead, fast mode posts p90 of 4348 ms — the lowest of the three (Crawl4AI 4754 ms, Firecrawl 6937 ms), at 91.8% scrape success of reachable URLs. Pick the mode that matches whether you're optimizing for coverage or for tail latency.

Search, briefly (a different benchmark)

Scraping and search are measured separately — the numbers above say nothing about search. Over a 100-query benchmark across 10 categories (benchmarks/triple-bench.ts, single point-in-time run):

• Average latency: fastCRW 880 ms · Firecrawl 954 ms · Tavily 2,000 ms
• Median: fastCRW 785 ms · Firecrawl 932 ms · Tavily 1,724 ms
• Latency wins: fastCRW 73 / 100 · Firecrawl 25 / 100 · Tavily 2 / 100
• Success rate: 100% / 100% / 100% across the board

Frozen phrasing: fastCRW search averaged 880 ms over a 100-query benchmark, with 73 of 100 latency wins over Firecrawl and Tavily.

When to self-host vs. run managed

Self-hosting is not always the right call, and pretending otherwise would be the marketing slop we are trying to avoid.

Self-host when:

• You want data locality — the pages you scrape never leave your infrastructure.
• You want zero per-scrape cost. The engine is AGPL-3.0, so it's $0 per 1,000 scrapes versus a hosted meter (Firecrawl's hosted cost is $0.83–5.33 per 1,000 scrapes across tiers, per competitor-prices.lock.md, verified 2026-05-18).
• You want to own the box, the renderer choice, and the update cadence.
• The AGPL-3.0 license fits your project. (If it does not, that is a genuine constraint, not a footnote.)

Run managed when:

• You would rather not operate the Chrome fallback yourself, or absorb that p90 tail on your own hardware.
• You need the managed LLM features — search answer/summarize and extract. These are metered in credits based on usage and are available on paid plans only; the free tier returns 402 for LLM calls, and there is no bring-your-own-key path. For pricing, see the live /pricing page rather than any number hard-coded in a blog post.
• You want someone else on the hook for uptime.

A common pattern: self-host the bulk scrape/crawl/map/search workload for cost and locality, and reach for managed only when you need the LLM-backed extraction on top.

Self-host configuration notes

A few things to know before you point fastCRW at production traffic, so you pick the right renderer and proxy setup up front.

PDF documents are supported. Point a PDF URL at /v1/scrape and it auto-routes through the built-in PDF parser to clean markdown, with extraction sandboxed in a separate container.

Anti-bot is built in. fastCRW ships a 12-signal block-detection classifier, user-agent rotation, and stealth browser fingerprints in the open core, wired into the auto escalation ladder (http → lightpanda → chrome → proxied chrome). Add your own external residential proxy via the proxy field, or opt into the managed cloud's residential-proxy egress tier, for the hardest targets.

Renderer footprint is a choice. The default lightweight Compose ships LightPanda for JS-challenge coverage at minimal footprint. For complex React/Vue/Angular apps with heavy client-side routing, opt into the chrome renderer variant — the same CDP-based renderer the auto escalation ladder uses on the managed cloud — for full SPA coverage.

Each request is stateless, and crawl status is polling-based via the crawl-status endpoint — you poll for progress rather than holding a persistent session. Expect minor response-field and error-envelope differences from Firecrawl when you port a client. robots.txt is respected by default.

These are the trade-offs of a single lean binary versus a heavier cluster — pick the renderer and proxy config that matches your target's difficulty and you get a small, cheap, self-hosted scraping API with the highest truth-recall of the three engines tested.

The short version

If you want a self-hosted web scraping API and the idea of running five containers for it makes you tired, fastCRW is the leanest serious option we know of: one ~8 MB binary, ~50 MB idle RAM, $0 per scrape under AGPL-3.0, and the highest truth-recall of three engines on Firecrawl's own benchmark.

Run it yourself, or read the full benchmark writeup and decide with the numbers in front of you.