Verify a Firecrawl Drop-In Replacement: Smoke Test

By the fastCRW team · Compatibility facts verified 2026-05-18 · benchmark figures from diagnose_3way.py, 2026-05-08 · Verify independently before you cut over.

Disclosure: we build fastCRW, so weight this accordingly. The point of this post is the opposite of a sales pitch — it hands you a harness to prove or disprove the compatibility claim on your own traffic, and it lists every documented place fastCRW diverges from Firecrawl so you find the gaps before production does.

Why verify a Firecrawl drop-in replacement instead of trusting the label

"Firecrawl-compatible" is a claim about the base API shape, not a promise about your specific code paths. fastCRW exposes a Firecrawl-compatible REST surface and is meant to be drop-in after a base-URL swap — but "the base shape matches" and "every field your pipeline reads is byte-identical" are two different statements. The first is what the vendor tests; the second is what your code actually depends on, and only you know which fields, which formats, and which error branches your pipeline touches.

So before you cut over a production workload, you want a compatibility smoke test: a small harness that replays your real requests against both APIs and asserts that the responses are shaped the way your code expects. This is verification, not migration. If you have not yet done the base-URL swap itself, that mechanic lives in the base-URL swap how-to and the API compatibility reference — this post assumes the swap is done and proves it holds.

What a smoke test actually proves

A smoke test does not prove "100% identical." It proves something more useful: that on the endpoints, formats, and inputs you use, the response status codes and top-level fields match closely enough that your downstream code does not break. A green run means your cutover is safe for the traffic you tested. A red run tells you exactly which divergence to adapt around — before it shows up as a null-pointer at 3am.

Building a compatibility smoke test harness

The harness has three moving parts: a representative request set, a dual-call runner, and a set of assertions. Keep it small and honest — twenty requests that mirror your real pipeline beat a thousand synthetic ones.

Replay representative requests against both APIs

Pick the requests that matter: the URLs your pipeline actually scrapes, the formats you actually ask for (markdown, html, links, json + jsonSchema), and the endpoints you actually call. Run each request twice — once against Firecrawl's base URL, once against fastCRW's — using the same SDK and the same request body. The only thing that changes between the two calls is api_url.

• Scrape: POST /v1/scrape with each format combination you depend on.
• Map: POST /v1/map against one representative site.
• Search: POST /v1/search with a query you would run in production.
• Crawl: POST /v1/crawl to get a job ID, then poll GET /v1/crawl/:id.

Assert status codes and top-level response fields

Start with the cheapest, highest-signal assertions: HTTP status code parity and the presence of the top-level fields your code reads. A typical Firecrawl-compatible scrape returns a success boolean and a data object containing the requested formats; assert those exist and have the right types on both backends.

Diff the returned markdown and JSON shapes

Content will never be byte-identical between two engines — different renderers extract slightly different markdown, and that is expected, not a failure. So do not diff content character-for-character. Instead assert structural parity: the markdown is non-empty and above a length floor, the links array is populated, and a json extraction validates against your jsonSchema with the required fields filled. For JSON extraction specifically, validate the returned object against the same schema you submitted — that is the real contract, not the surrounding envelope.

The divergence matrix to check explicitly

This is the part most teams skip and later regret. fastCRW documents exactly where it diverges from Firecrawl, so your harness should assert against this list directly rather than discovering it in production.

Minor field-name and error-envelope differences

Response field names and error envelopes have minor divergence from Firecrawl. On the happy path most pipelines never notice, but error-handling code that reads a specific error field by name is the classic break point. Add an assertion that deliberately triggers an error (a bad URL, a missing required field) on both backends and confirms your handler still extracts a usable message from each envelope.

Screenshot output (supported on v2)

Screenshot output is supported on the v2 scrape API. A request for formats: ["screenshot"] returns data.screenshot as a base64 PNG data URL (data:image/png;base64,...); use formats: ["screenshot@fullPage"] for a full-page capture. This mirrors Firecrawl's contract — the field is the same opaque string both SDKs already expect — so add a screenshot request to your harness and assert that data.screenshot is a non-empty data URL on both backends.

/v1/agent and persistent sessions

fastCRW has no /v1/agent (Spark-style) endpoint and no persistent interactive session — it's stateless per request, plus the async /v1/crawl job. Multi-URL extraction and deep research are both covered: /v1/extract accepts up to 50 URLs per request, and /v1/search/research/papers plus /v1/search/research/papers cover deep-research as composable primitives. If your current Firecrawl integration calls /v1/agent specifically, that call is a hard fail and should be flagged loudly by the harness.

Reading and acting on test results

Bucket every assertion result into one of three outcomes. The bucket decides what you do next.

Pass: behaviour matches on your endpoints

Status codes, top-level fields, and content shapes match on every endpoint and format you exercised. This is the common outcome for scrape/crawl/map/search pipelines, and it is supported by the engine's reliability profile: on Firecrawl's own public dataset, fastCRW recorded 0 thrown errors paired with 91.8% scrape-success of reachable URLs (diagnose_3way.py, 819 labeled URLs, 2026-05-08). A pass means you can cut over the tested traffic with confidence.

Soft-fail: a divergence you can adapt around

A field is named differently, or an error envelope reads slightly differently, but your code can be adjusted in a few lines — read the new field name, normalize the envelope, done. These are the field-name and error-shape divergences. Patch your client, re-run the harness, and confirm it goes green.

Hard-fail: a capability fastCRW does not have

You asked for an agent endpoint, and the capability simply is not there. No client patch fixes this — you either remove the dependency or rearchitect around fastCRW's composable primitives (search + scrape/crawl with your own orchestration, or the built-in /v1/search/research/papers API). See the head-to-head comparison for where each tool wins.

A safe rollback procedure

Never cut over without a rollback path. Because the only thing that changed is the base URL, rollback is genuinely a one-line revert — but only if you set it up that way before you flip.

Keep the old base URL behind a flag

Put the API base URL behind an environment variable or config flag rather than hard-coding it. Cutover is then flipping the flag from the Firecrawl URL to the fastCRW URL; rollback is flipping it back. No code change, no redeploy of logic, no fork.

One-line revert and re-run the harness

If production telemetry shows a divergence the smoke test missed, revert the flag and immediately re-run the harness with the failing request added to the set. The harness is not a one-time gate — it is the regression suite that proves the next attempt is fixed. Every production surprise becomes a permanent assertion.

Self-host or managed — the same API on both

One detail that simplifies rollback and staging: fastCRW exposes the same API whether you run the self-hosted single binary or the managed cloud at fastcrw.com. You can run the smoke test against a local self-hosted instance for free, prove compatibility there, then point the same harness at managed cloud — or vice versa — without rewriting a single assertion. Compare the cost side of that decision on the pricing page once compatibility is proven.

Sources

• fastCRW open-core README — endpoint table and renderer aliases: github.com/us/crw
• Firecrawl API reference (for the base shape you are matching): docs.firecrawl.dev (verified 2026-05-18)
• Scrape benchmark of record — diagnose_3way.py, Firecrawl public dataset, 819 labeled URLs, 2026-05-08

Related: The Firecrawl SDK base-URL swap · Firecrawl API compatibility · fastCRW vs Firecrawl