What is data egress in web scraping?

Data egress is any data that leaves your network as part of a scraping pipeline beyond the requests to the target sites themselves. With a cloud scraping API, three things egress to the vendor: the target URLs you fetch, the scraped content, and any LLM extraction prompts and schemas. Zero-egress scraping means none of those reach a third party — the engine runs inside your own boundary so the bytes never have a vendor hop.

How do I scrape with zero data egress?

Run the scraping engine inside your own network instead of calling a hosted API. fastCRW's engine is a single static Rust binary (AGPL-3.0) you can run in your VPC or on-prem; keep any LLM extraction on a local model inside your boundary; and use a firewall egress allow-list so the host can only reach the target domains. There is no vendor callback, so the scraped content never leaves.

Does a self-hosted scraper guarantee no data leaves my network?

Self-hosting the engine removes the scraping vendor from the path, but it does not by itself stop other leaks. The collection host still needs egress to the target sites, and if you use a cloud LLM for extraction, that provider sees the prompt and content. For true zero egress you pair a self-hosted engine with an egress allow-list and a local model, then prove it with network flow logs from the host you control.

Can I keep LLM extraction inside my own infrastructure?

Yes. Self-host the engine and run a local model for the extraction step, so the prompt and page content never leave your network at all — that is the truly air-gapped path. Any cloud model you call instead would still see the prompt and content. On the managed cloud, LLM-based JSON extraction is a managed feature available on paid plans.

Zero Data Egress Scraping for Compliance

By the fastCRW team · Comparisons and pricing verified 2026-05-18 · Verify independently before relying on it.

Disclosure: we build fastCRW. This is a vendor-authored engineering post, so weight the framing accordingly — but the architecture below is provider-neutral, and we name the cases where a managed cloud (including ours) is the wrong tool for a zero-egress requirement.

What zero data egress means in a scraping pipeline

Zero data egress scraping means that when you collect web data, nothing about that collection leaves your network except the requests to the target sites themselves. No third-party scraping vendor sees the URLs you fetch, the content you extract, or the prompts you use to structure it. For a security or compliance engineer, "egress" is not a vibe — it is a measurable property of your network: which destinations receive bytes, and what those bytes contain.

The reason this matters is that a conventional cloud scraping API inverts the trust model. You send it a URL, it fetches the page on its own infrastructure, runs any extraction, and sends the result back to you. That is convenient, but it means the vendor's servers are a mandatory hop for every byte of your pipeline.

The three leaks: target URLs, scraped content, extraction prompts

A scraping pipeline has three distinct data flows that can egress to a third party, and teams often plug one while leaving the others open:

Target URLs. The list of pages you scrape is itself sensitive. If you are enriching a customer list, monitoring competitors, or pulling internal/intranet sources, the URL set reveals your intent before a single page is fetched.
Scraped content. The extracted markdown or HTML — the actual data — transits the vendor and may be logged or retained.
Extraction prompts and schemas. If you use LLM extraction (formats: ["json"] with a jsonSchema), your prompt and schema describe exactly what you are looking for, and the LLM provider sees both the prompt and the page content.

Where a cloud scraping API sends your data

With a hosted API, all three leaks are live by default. Firecrawl, for example, is cloud-only; its zero-data-retention mode is gated behind the Enterprise tier rather than being the default posture. That is a reasonable product decision, but it means the privacy property you need is a contract term, not an architectural guarantee you can verify yourself.

Why egress is a compliance and exfiltration concern

Two audiences care about this for different reasons. Compliance owners care because GDPR, HIPAA, and sector data-residency rules treat any third-party processor of personal or regulated data as something you must contract for, document, and audit. Security teams care because every outbound destination is an exfiltration path: a scraping API that sees your data is, from a threat-model standpoint, a place your data can leak from even if no one intends it to.

The zero-egress architecture

The only way to get true zero egress is to run the scraping engine inside your own boundary, so the bytes never have a vendor hop to begin with. The architecture has three moving parts.

Run the open-core engine inside your VPC or on-prem

fastCRW's engine is a single static Rust binary — no Redis, no Node.js, no containers required — released under AGPL-3.0 (github.com/us/crw). Its structural footprint is a single ~8 MB binary in one container, versus a multi-service hosted stack reported at ~2–3 GB across 5 containers (README "Structural footprint", framed as a structural fact, not a benchmark). That small, self-contained shape is precisely what makes it droppable inside a VPC, an on-prem subnet, or a locked-down VM. You run it; you own the network it sits on.

Keep LLM extraction inside your own boundary

Scraping markdown is one thing; structuring it with an LLM is the leak most people forget. The strongest posture is to self-host the engine and run a local model for the extraction step, so the prompt, schema, and page content never leave your network. Be precise about what that buys you: any cloud model you call would still see the prompt and content, so for a hard zero-egress requirement that includes the model, you must run a local model. On the managed cloud, LLM-based JSON extraction is a managed feature available on paid plans (managed search synthesis uses a managed LLM too) — so the truly air-gapped path is self-host plus your own local inference.

Egress allow-lists: only the target sites, nothing back to a vendor

With the engine local, your firewall does the enforcement. Configure an egress allow-list so the collection host can reach the target domains (and your own local model endpoint if you run extraction) and nothing else. There is no vendor callback URL, no telemetry endpoint, no result-upload step — because the result never left. That allow-list is also your audit evidence (see below).

What you trade away for zero egress

Zero egress is a real architectural win, not a free one. State the trade-offs plainly or you will get surprised in production.

Self-hosted anti-bot: built-in detection, bring your own proxy pool

fastCRW's open core ships 12-signal block detection, user-agent rotation, and stealth fingerprints — that anti-bot logic runs the same whether you self-host or use managed cloud. What differs is the proxy pool: on managed cloud, paid-plan traffic is routed through fastCRW's residential-proxy pool automatically; self-hosted, you configure your own proxy credentials. For a strict zero-egress requirement, routing through any third-party proxy pool reintroduces an egress hop, so the zero-egress story is strongest for sources you can reach directly: internal apps, intranet pages, partner portals, and public sites that do not require proxy depth.

Stateless engine: you own logging and retention

fastCRW is stateless per request — there is no persistent session or server-side history. For a privacy posture that is mostly a feature: the engine is not accumulating a copy of your data. But it means scheduling, audit logging, and retention are your responsibility to build around it. That is the correct division of labor for a zero-egress design, but it is work you own.

Screenshots run through your own boundary too

Screenshots are supported: a request for formats: ["screenshot"] returns a CDP-captured base64 PNG data URL. On the zero-egress path this renders entirely on your self-hosted engine, so the image never reaches a third party either.

Proving zero egress to auditors

"We self-host" is a claim. Auditors want evidence. The good news is that a local-first architecture produces evidence that is far easier to gather than a vendor attestation.

Network policy and traffic capture as evidence

Your egress allow-list is the primary control, and a packet capture or flow log from the collection host is the proof. You can demonstrate, with your own data, that outbound connections went only to the target domains (and your LLM endpoint), and never to a scraping vendor. That is a stronger statement than "the vendor says they don't retain data," because it is verifiable on infrastructure you control.

Source-available AGPL-3.0 code review

Because the engine is AGPL-3.0 and the source is public, your security team can review exactly what the binary does — there is no hidden phone-home you have to take on faith. For procurement teams that need to assert what code is running in a regulated environment, source-available beats a closed binary every time.

Cost: $0 per 1,000 self-hosted scrapes vs metered cloud

The compliance win comes with a cost win. Self-hosting the AGPL-3.0 engine is $0 per 1,000 scrapes — you pay only for your own server. A hosted API meters every request: Firecrawl's hosted pricing works out to roughly $0.83–5.33 per 1,000 scrapes across its tiers (competitor-prices.lock.md, verified 2026-05-18). For a high-volume pipeline that runs forever, the local path removes both the egress and the per-page meter at the same time. See /pricing for the managed tiers if you do mix in cloud.

Hybrid: zero-egress local plus managed scale

Most real teams do not have a uniform privacy requirement. Some workloads are regulated; some are bulk public data where a vendor hop is acceptable under policy. The practical pattern is to run both on one compatible API.

Same API for sensitive and bulk workloads

fastCRW speaks a Firecrawl-compatible REST API on both the self-hosted engine and the managed cloud, so routing a job to "local zero-egress" versus "managed scale" is a base-URL swap, not a code fork. Sensitive jobs hit the in-boundary binary; bulk, non-sensitive jobs can hit the managed cloud (or stay local for cost). The compatible surface is what keeps this from becoming two pipelines.

When a managed call is acceptable under policy

Write the rule down. A managed call is fine when the target URLs are public and non-revealing, the content is not regulated, and no extraction prompt would leak intent — for example, scraping public documentation for a general index. It is not fine the moment any of the three leaks touches regulated or sensitive data. Encode that as routing logic, not tribal knowledge, and your zero-egress guarantee survives contact with a deadline.

Sources

fastCRW canonical fact sheet — product identity, structural footprint, cost, capabilities
Self-hosted vs hosted cost: marketing/competitor-prices.lock.md (verified 2026-05-18)
fastCRW repo (AGPL-3.0, single binary, endpoint table): github.com/us/crw
Firecrawl docs/pricing (cloud-only, Enterprise zero-data-retention): docs.firecrawl.dev (verified 2026-05-18)