By the fastCRW team · Architecture and pricing facts verified 2026-05-18 · Verify independently before relying on this for a compliance decision.
Disclosure: we build fastCRW. This is a vendor-authored piece, and on the one question that matters most here — "is this tool a compliance certificate?" — the honest answer is no, for fastCRW or any other scraper. We say so plainly below, and we name where managed cloud APIs genuinely fit better.
Why a privacy-first web scraping API is hard to find for regulated work
If you work in health, finance, legal, or government, you have probably hit the wall already: you need web data, but the obvious tools all want to route your queries and your results through someone else's cloud. A normal scraping API is, architecturally, a data processor you have invited into the middle of your pipeline. A privacy-first web scraping API inverts that — the engine runs inside your boundary, and the only thing that crosses the network is the request to the public site you were going to fetch anyway. That distinction is the whole article.
Every scrape routes target URLs and content through the vendor
With a cloud-only scraping API, three things leave your network on every call: the target URL (which can itself be sensitive — an internal-facing portal, a patient-portal domain, a counterparty's filing), the scraped content that comes back, and, if you use AI extraction, the extraction prompt and schema. For a regulated workload, each of those is a disclosure to a third party that your data-protection officer has to account for.
Data residency, retention, and processor agreements
Once a vendor sees your data, you inherit their region, their retention window, and their sub-processor list. Under GDPR that means a controller-processor agreement and a lawful basis for the transfer; under a HIPAA workflow it means a Business Associate Agreement (BAA) and an accounting of where protected health information lands. None of that is impossible — it is just paperwork and risk you would not have if the data never left.
Where managed APIs gate privacy behind Enterprise tiers
The common pattern is that the privacy control you actually need — zero data retention — exists, but only on the Enterprise tier. Firecrawl, the category reference whose API shape fastCRW mirrors, offers zero-data-retention as an Enterprise-only feature. That is a legitimate business model, but it means the small or mid-size regulated team often can't buy the one guarantee they need at a price they can justify.
What "privacy-first" actually requires
Marketing uses "private" loosely. For a regulated pipeline it has a concrete, testable definition.
Zero third-party egress of queries and results
The strongest version is structural: the scraping engine itself runs inside your VPC or on-prem, so neither the target URL nor the returned content is ever sent to a vendor. The engine reaches out to the public site, the bytes come back to your machine, and that is the end of the trip. This is the property a cloud-only API cannot offer without a contractual add-on, because the architecture sends your data to them by design.
Self-hosting any LLM extraction step
If you extract structured JSON, an LLM is in the loop, and that is a second egress point most teams forget. Privacy-first means you decide which model sees the page — and the structural answer is to self-host the engine and run your own local inference, so the extraction never leaves your boundary at all. (On the managed cloud, LLM-based JSON extraction is a managed feature on paid plans; for a hard zero-egress requirement, self-host plus a local model is the path.)
Auditable, self-hosted infrastructure you own
"Trust us" is not a control. Source-available code you can review, plus infrastructure you run and can point a network capture at, is. fastCRW's engine is AGPL-3.0 and the repo is public at github.com/us/crw, so your security team can read exactly what the binary does before it touches a single sensitive URL.
Building a privacy-first scraping pipeline
Here is the concrete shape of a pipeline that keeps data inside your boundary.
Self-host the open-core engine inside your boundary
fastCRW's engine is a single static Rust binary — no Redis, no Node, no container sprawl — and ships as a single ~8 MB image / 1 container (a structural fact from the README, not a benchmark). You run it on a box you control. Because it speaks a Firecrawl-compatible REST surface (/v1/scrape, /v1/crawl, /v1/map, /v1/search), your code talks to it exactly as it would to a hosted API — the only difference is the base URL points at your own host. See self-host vs managed scraping for the deployment trade-offs.
Keep LLM extraction inside your own boundary
The strongest privacy posture is to self-host the engine and run a local model for the extraction step, so the prompt, schema, and page content never leave your network. When you pass formats: ["json"] with a jsonSchema, the extraction runs against the model you point the engine at — keep that model inside your boundary and the structuring step has no egress at all. (Note: on the managed cloud, LLM-based JSON extraction is a managed feature available on paid plans; the truly air-gapped path is self-host plus your own local inference — more on the managed limits below.)
Respecting robots.txt and lawful-access controls
Privacy-first cuts both ways: you owe the same respect to the sites you fetch. fastCRW honors robots.txt by default and only allows an override when the caller has the legal right to do so. For regulated work, that default is a feature — it keeps your collection inside the lawful-access line your counsel drew.
Compliance is the deployer's job, not the tool's
This is the section a vendor is tempted to skip. We won't.
fastCRW provides data-residency primitives, not a certification
Self-hosting an open-core engine gives you the primitives to meet GDPR or HIPAA obligations — data stays in your region, on your hardware, under your control. It does not give you a certification. The architecture removes the egress; you still have to do the access controls, encryption, logging, and the actual risk assessment.
No BAA, no compliance certification — you own the obligation
fastCRW is a tool, not a BAA-covered or SOC-2-certified processor for your specific deployment. If your workflow handles PHI and needs a BAA, no scraper binary can sign one for you — the obligation lives with you as the deployer. The structural advantage is precisely that you don't need a third party in scope, because no third party is in the data path.
Stateless engine means you own audit logging and retention
fastCRW is stateless per request — it does not persist your jobs, results, or sessions. For audit purposes that is a double-edged property: nothing lingers in the engine to leak, but you must capture your own audit log and define your own retention. Wire request/response logging into your existing SIEM the same way you would for any internal service.
What to plan around
A privacy-first architecture has real trade-offs worth planning for up front.
Anti-bot and residential proxy coverage is built in
fastCRW ships 12-signal block detection, user-agent rotation, stealth fingerprints, and proxy rotation through a residential-proxy egress tier in the open core, so hardened targets behind common bot defenses are covered without a third-party anti-bot service in your data path. Screenshots are supported too: pass formats: ["screenshot"] and get a base64 PNG data URL back.
Stateless requests — you own session and audit logging
Because the engine holds no session state, anything that needs persistence — login sessions across multi-step flows, job history, audit trails — is your responsibility to orchestrate around it. That is the price of an engine that keeps nothing.
Managed cloud when you need it, without losing control
Not every job is sensitive. The point of an open-core model is that you don't have to choose one path forever.
Free credits to start, managed LLM on paid plans
The managed cloud starts free (500 one-time lifetime credits — not a monthly meter), and LLM features such as managed JSON extraction and answer synthesis run on paid plans; the Free plan has no LLM features. So you can run sensitive jobs on the self-hosted engine and burst non-sensitive volume to the managed cloud. Managed pricing lives on the live pricing page — don't trust a hard-coded table, including this one.
Same API surface as the self-hosted engine
The managed cloud and the self-hosted binary expose the same Firecrawl-compatible API. Routing a job from "in our boundary" to "in the cloud" is a base-URL change, not a rewrite. That means your privacy posture is a runtime decision per workload, not a vendor lock-in you committed to at design time. For the broader privacy rationale, see local-first scraping and data privacy and the licensing implications in AGPL-3.0 for SaaS, explained.
Sources
- fastCRW canonical facts: engine, AGPL-3.0 license, footprint — github.com/us/crw
- Firecrawl zero-data-retention as an Enterprise-only feature: docs.firecrawl.dev (verified 2026-05-18)
- GDPR controller/processor framework: gdpr.eu; HIPAA BAA guidance: hhs.gov/hipaa
Related: Local-first scraping and data privacy · Self-host vs managed scraping · AGPL-3.0 for SaaS