Best Firecrawl Alternative for Self-Hosting

Here is the recurring problem in one sentence: self-hosting a web scraping API should not mean standing up a five-container cluster and feeding it a few gigabytes of RAM before you have scraped a single page.

If you have tried to self-host Firecrawl, you know the shape of it. Lovely engineering, genuinely — but the footprint is built for their hosted platform, not for a box you own. fastCRW is the other end of that tradeoff: a self-hosted web scraping API that ships as a single ~8 MB Rust binary, idles at roughly 50 MB of RAM on a $5 VPS, and speaks a drop-in-compatible API so your existing client code mostly keeps working.

Two things make it worth a look before you reach for a cluster:

1. It is more open and more local. The engine is AGPL-3.0. You run the binary, your data never leaves your box, and there is no per-scrape meter.
2. It is accurate where it counts. On Firecrawl's own public benchmark, fastCRW recovered more of the labeled content than either Firecrawl or Crawl4AI. Numbers and provenance below — we do not ask you to take that on faith.

This post covers how self-hosting actually works, what the footprint and accuracy numbers really say (including the parts that do not flatter us), and when you should self-host versus let someone else run it.

The footprint: what "self-hosted" usually costs, and what it costs here

These are structural facts from the README (§Structural footprint), not benchmark claims — but they are the whole reason this post exists.

For comparison, a Crawl4AI/Playwright-style setup is cited at 300 MB+ idle RAM and roughly 200–300 MB per worker (how-we-built-fastcrw). None of these are wrong tools — Playwright is excellent, and Crawl4AI is a pleasure inside a Python notebook. But if your goal is "a scraping API on a small box that I never have to babysit," an 8 MB binary that idles at ~50 MB is a different category of thing than 5 containers and a couple of gigabytes.

The catch, stated up front: the default binary uses LightPanda, an experimental Zig-based browser that implements a subset of the W3C APIs. It is not Chromium. That keeps the image tiny, and it handles server-rendered and server-hydrated pages well. For heavy client-side apps it has real limits — covered in the honest-gaps section. There is an opt-in Chrome Compose variant (~500 MB image, ~1 GB resident) for when you need a full browser. You opt into that weight only when a page actually demands it.

How self-hosting works

The moving parts are deliberately few.

1. Run the binary. The default Compose file brings up one container running the binary with the LightPanda renderer. No queue service, no separate browser pool, no Redis-shaped dependency to keep alive.

2. Call a Firecrawl-shaped API. Your existing scrape/crawl/map/search calls port over with minor changes. A basic scrape:

curl -X POST http://localhost:3002/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/article",
    "formats": ["markdown"]
  }'

3. Pick a renderer when you need to. The engine selects a renderer automatically, but you can force one. For a client-heavy page that comes back thin under LightPanda, force Chrome (this requires the opt-in Chrome variant):

curl -X POST http://localhost:3002/v1/scrape \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://app.example.com/dashboard",
    "formats": ["markdown"],
    "renderer": "chrome"
  }'

4. There is no per-scrape bill. When you self-host the AGPL-3.0 engine, scrapes are free — you pay only for your server. The credit costs that apply on the managed service (scrape = 1 credit as of 2026-06-24, crawl = 1 per page, search = 1 per query, map = 1, an MCP browse session = 1, and any formats: ["json"]/extract request = 5, per the README §Renderer selection) are the managed metering model, not a toll you pay when you run the binary yourself.

That is the entire mental model: one process, a familiar API, renderer control when a page fights back, and no meter.

The accuracy story, on Firecrawl's own benchmark

This is the part that should decide whether an 8 MB binary is a serious tool or just a small one. We ran all three engines against Firecrawl's public scrape-content-dataset-v1 (1,000 URLs, 819 of them labeled), using diagnose_3way.py, in a single run of 3,000 requests on 2026-05-08. Every number in this section comes from that run.

Truth-recall, of the 819 labeled URLs — how much of the content that was supposed to be there did each engine actually recover:

• fastCRW: 63.74% (522) — highest of the three
• Crawl4AI: 59.95% (491)
• Firecrawl: 56.04% (459)

That is +3.79 pp over Crawl4AI and +7.70 pp over Firecrawl on Firecrawl's own dataset. Recall is the metric we optimize for, because a scrape that returns HTTP 200 with half the article missing is a silent failure — it looks like success and poisons whatever you feed it into.

Scrape-success, of all 1,000 URLs — did a request come back with usable content at all:

• Firecrawl: 89.7% (897) — highest
• fastCRW: 87.7% (877)
• Crawl4AI: 83.5% (835)

We report this one honestly too: Firecrawl edges us on raw success rate. Different question, different winner. Recall asks "how complete," success asks "did anything come back." We win the first, they win the second, and pairing them is the only fair way to read either. (The 0 thrown errors across all 3,000 requests for every engine is real, but it means nothing without that 87.7% next to it — "zero errors" alone would overstate the case, since a thin-but-successful response throws no error.)

The latency tradeoff, in full

Here is where we do not get to pick the flattering number. Scrape latency, same run:

Read that honestly and it tells a real story. At the median, fastCRW is the fastest of the three — effectively tied with Crawl4AI, 2 ms apart. But our p90 is the worst by a wide margin: 14157 ms against Crawl4AI's 4754 ms.

That tail is not a bug we are hiding — it is causal. When a page fails the cheap renderers, fastCRW falls back to a Chrome-stealth path to recover it rather than giving up. Those recovered pages are exactly the ones that push the recall number up, and they are exactly the ones that show up in the slow tail. The engines with a tidier p90 got there partly by not chasing the hard pages. You are trading a slow 10% for a higher truth-recall. Whether that trade is right depends entirely on your workload, which is why we publish p50, p90, and p99 every time and never a single averaged latency — an average would hide precisely the thing you need to know.

If your workload cannot tolerate that tail, there is a fast mode: its p90 is 4348 ms, the lowest of the three, at 91.8% scrape success of reachable URLs (how-we-built-fastcrw). Faster tail, slightly lower coverage — the same trade, dialed the other way.

Search, briefly (a different benchmark)

Scraping and search are measured separately — the numbers above say nothing about search. Over a 100-query benchmark across 10 categories (benchmarks/triple-bench.ts, single point-in-time run):

• Average latency: fastCRW 880 ms · Firecrawl 954 ms · Tavily 2,000 ms
• Median: fastCRW 785 ms · Firecrawl 932 ms · Tavily 1,724 ms
• p95: fastCRW 1,433 ms · Firecrawl 1,343 ms · Tavily 3,534 ms
• Latency wins: fastCRW 73 / 100 · Firecrawl 25 / 100 · Tavily 2 / 100
• Success rate: 100% / 100% / 100% across the board

Frozen phrasing: fastCRW search averaged 880 ms over a 100-query benchmark, with 73 of 100 latency wins. Firecrawl takes the p95 by a hair; we take the average, the median, and the head-to-head win count.

When to self-host vs. run managed

Self-hosting is not always the right call, and pretending otherwise would be the marketing slop we are trying to avoid.

Self-host when:

• You want data locality — the pages you scrape never leave your infrastructure.
• You want zero per-scrape cost. The engine is AGPL-3.0, so it's $0 per 1,000 scrapes versus a hosted meter (Firecrawl's hosted cost is $0.83–5.33 per 1,000 scrapes across tiers, per competitor-prices.lock.md, verified 2026-05-18).
• You want to own the box, the renderer choice, and the update cadence.
• The AGPL-3.0 license fits your project. (If it does not, that is a genuine constraint, not a footnote.)

Run managed when:

• You would rather not operate the Chrome fallback yourself, or absorb that p90 tail on your own hardware.
• You need the managed LLM features — search answer/summarize and extract. These are metered in credits based on usage and are available on paid plans only; the free tier returns 402 for LLM calls, and there is no bring-your-own-key path. For pricing, see the live /pricing page rather than any number hard-coded in a blog post.
• You want someone else on the hook for uptime.

A common pattern: self-host the bulk scrape/crawl/map/search workload for cost and locality, and reach for managed only when you need the LLM-backed extraction on top.

The honest gaps

A whole post on this site is titled "Where CRW Still Falls Short," and this section is in that spirit. If you self-host, these are the things that will bite you, so here they are up front with severity and workarounds.

No PDF / DOCX / document parsing. HTML pages only. Point a PDF URL at /v1/scrape and you get empty or an error. Route documents to a dedicated parser (pdfplumber or similar) before or alongside fastCRW. PDF text extraction is on the roadmap but not this cycle; DOCX is further out.

Anti-bot is not best-in-class. You get realistic user agents, header mimicry, cookie handling, and delays — fine for the roughly 80% of public content that is not aggressively defended. Against Cloudflare Enterprise, PerimeterX, DataDome, or Akamai Bot Manager, you will get blocked more often than a specialized service would. There is no Fire-engine-style anti-bot layer. Workaround: bring your own external residential proxy via the proxy field. CAPTCHA solving is not planned; built-in proxy rotation is being evaluated.

SPA coverage via LightPanda is inconsistent. This is the highest-priority core gap and it is under active development. The default LightPanda renderer handles server-hydrated pages well, but complex React/Vue/Angular apps with heavy client-side routing and lazy loading can come back thin. Detection heuristic: under 200 words on a page you know is content-rich. Workaround: force the chrome renderer (opt-in variant), or fall back to Playwright/Firecrawl for that subset.

It's stateless, and crawl progress is polling-only. Each request is independent — no persistent session, no WebSocket or SSE streaming of crawl progress. You poll for status. There are also minor response-field and error-envelope divergences from Firecrawl to expect when you port a client. robots.txt is respected by default.

None of these are dealbreakers for the common case — a small, cheap, self-hosted scraping API that pulls HTML content with high recall. They are exactly the cases where you should keep a heavier tool in your back pocket, and knowing that in advance is the whole point.

The short version

If you want a self-hosted web scraping API and the idea of running five containers for it makes you tired, fastCRW is the leanest serious option we know of: one ~8 MB binary, ~50 MB idle RAM, $0 per scrape under AGPL-3.0, and the highest truth-recall of three engines on Firecrawl's own benchmark. It is slower in the tail, it does not read PDFs, and it will lose to a specialist on the meanest anti-bot sites. Everywhere in between, an 8 MB binary does the job.

Run it yourself, or read the full benchmark writeup and decide with the numbers in front of you.