The Real Cost of Web Scraping at Scale: A TCO Model (2026)

By the fastCRW team · Engineering/economics deep-dive · Last reviewed 2026-01-01

Disclosure: fastCRW is one option in the model below and we build it. The TCO framework is general — substitute your own vendor's current numbers; the structure is what matters.

Why per-page price is the wrong unit

Teams budget scraping with a single number — "$X per 1,000 pages" — and are then surprised by the invoice. The surprise is structural: at scale, the per-page API charge is often a minority of total cost. A real TCO model has five line items, and any one of them can dominate depending on your target mix. This post builds that model so you can compute your break-even instead of trusting a pricing page or a "self-host is free" slogan (both are wrong at scale).

The five TCO line items

1. Base extraction cost — the advertised per-page or per-credit charge, or the compute to run a self-hosted engine.
2. Multiplier tax — output-format and rendering-mode multipliers on metered APIs (JSON/enhanced/stealth modes stacking to several credits/page).
3. Retry tax — billed attempts on failed pages. Scales with how hostile your targets are.
4. Proxy spend — residential/mobile IP cost for protected sites. Often the largest single line at scale and frequently invisible until you hit Cloudflare.
5. Engineer-time — the unpriced line. Operating self-host, debugging coverage gaps, on-call for scrape failures, building retry/queue logic. Denominated in your scarcest resource.

"Hosted is expensive" usually means lines 1–3 are visible and painful. "Self-host is free" usually means lines 4–5 were ignored. A correct decision prices all five.

Worked model: 1,000,000 pages/month

Assume a production RAG/agent corpus: 1M pages/month, JSON output required, 20% of targets behind anti-bot needing proxy + retries.

Option A — Metered hosted API (multiplier model)

• Base: 1M pages. Naive read of the pricing page: one large tier.
• Multiplier tax: JSON + enhanced mode pushes effective credits/page well above 1 — frequently 3–9× for RAG-shaped requests.
• Retry tax: 20% protected, with retries, inflates billed requests further.
• Extraction surcharge: if structured extraction is a separate subscription, add a fixed monthly floor on top.
• Effective: the "1M page plan" routinely behaves like a 2.5–4M page plan in dollars. This is the single biggest source of "the bill detonated" complaints in 2026.

Option B — Flat metered API (1 page = 1 credit on every renderer)

• Base: 1M pages = 1M credits. JSON costs the same as markdown. No separate extraction meter.
• Multiplier tax: $0 (the model has none).
• Retry tax: you still pay for retries you issue, but a faster/leaner engine that succeeds more often on the first try reduces retry volume — and you control retry policy.
• Proxy spend: covered inside the managed price for the hosted path.
• Effective: the naive budget is close to the real budget. This is fastCRW Cloud's model and the entire point of pricing legibility.

Option C — Self-host an open-core engine

• Base: compute. A small Rust binary with a low idle footprint runs 1M pages/month on a small VPS — order of magnitude $10–50/month of compute depending on concurrency and target latency. License: $0 (AGPL-3.0).
• Multiplier tax: $0.
• Retry tax: your compute, not a per-request charge — effectively free at the margin.
• Proxy spend: you now own this line. For the 20% protected fraction you need residential/mobile IPs. This is the dominant cost of self-host at scale and is frequently hundreds to thousands per month for hostile target sets — far more than the compute.
• Engineer-time: building/operating retry, concurrency, monitoring, proxy rotation, and being on-call. For a heavyweight stack (Redis + 5 services + multi-GB RAM) this is large; for a single 6MB binary it is small but not zero.
• Effective: compute is trivial; the real cost is proxy spend + engineer-time. For non-hostile corpora self-host is dramatically the cheapest. For hostile corpora the proxy line can erase the savings versus a managed price that includes a proxy network.

The break-even logic

The decision is not "hosted vs self-host" globally — it's per-workload, driven by two variables:

• Volume. Higher volume favors self-host on lines 1–3 (compute scales sublinearly with a lean engine; metered cost scales linearly with multipliers).
• Target hostility. Higher hostility favors the managed path on line 4 (proxy spend), because amortized proxy networks beat DIY proxy contracts for most teams.

So: high volume + low hostility → self-host wins decisively (compute is pennies, no proxy line). Low volume + high hostility → managed wins (proxy network amortized for you, low volume doesn't justify the ops). The interesting case is high volume + high hostility, where the answer is often both: self-host the easy 80%, route the hostile 20% to a managed proxy-backed path — which only works cleanly if both paths share one API.

Why the same-engine-both-ways property is a cost lever, not just a feature

Most cost models assume you pick one tool and eat its cost curve. fastCRW's design makes the hybrid split practical: the AGPL-3.0 self-host engine and the Managed Cloud expose an identical API, so routing the cheap bulk to your own boxes and the hostile fraction to the Cloud is a base-URL decision per request class, not two integrations. That converts the break-even from a one-time vendor lock-in into a continuously tunable dial — and it makes the $0 self-host floor a permanent ceiling on the managed price, because you can always move workload back if the economics change.

The retry tax deserves its own paragraph

Retries are the most underestimated line. A 10,000-page crawl with a third of pages failing and retrying doesn't cost 10,000 units — it costs ~13,000+, silently, and the failures often produce silent gaps in a RAG corpus (so you pay more and get worse data). Two defenses: (1) a faster, higher-coverage engine reduces first-attempt failures, shrinking the tax at the source; (2) a flat pricing model caps the damage because retries don't carry a multiplier. The worst combination is a multiplier model on hostile targets — there, retries multiply a number that was already multiplied.

The TCO checklist

1. Compute effective credits/page for your required output format, not the headline rate.
2. Estimate your protected-target fraction and model the retry tax on it.
3. Price proxy spend explicitly — it's the dominant self-host cost at scale and often hidden in managed pricing.
4. Put a dollar figure on engineer-time for operating the cheapest option (small for one binary, large for a 5-service stack).
5. Decide per workload, not globally; consider splitting easy vs. hostile traffic across self-host and managed with one API.

Bottom line

The real cost of web scraping at scale is rarely the per-page price — it's the multiplier tax, the retry tax, proxy spend, and engineer-time. Metered multiplier models detonate on exactly the high-volume RAG workloads people run; "free self-host" ignores proxy and ops cost. The lowest-TCO posture for most teams at scale is: self-host a lean open-core engine for the non-hostile majority, use a managed proxy-backed path for the hostile minority, and keep one API across both so the break-even stays tunable. That's the architecture fastCRW is built for, and the reason the model — not the sticker price — should drive the decision.

Model it yourself

docker compose up   # the $0-license / compute-only line item

Managed Cloud for the hostile fraction: one-time lifetime 500 free credits, no card. fastcrw.com · GitHub

Related: Cheapest web scraping API · Self-host vs managed scraping · Anti-bot & proxies overview