Scraping Latency Explained: Where the Milliseconds Actually Go

By the fastCRW team · Engineering deep-dive · Last reviewed 2026-01-01

Disclosure: fastCRW is built around the latency model below and we maintain it. Any comparative speed framing references our public, reproducible 1,000-URL benchmark — run it yourself.

Why a scrape that "feels slow" is an architecture problem

"The scrape is slow" is usually diagnosed as "the site is slow." Sometimes true — but in agentic workloads the dominant slowness is frequently the scraper's own architecture, not the network. To reason about it you need the actual latency budget: a single scrape decomposes into six segments, and an engine's design decides which ones it pays. Let's trace one request from "agent calls the tool" to "clean markdown returned."

The six latency segments

1. Request admission — time from your call hitting the scraper to it actually starting work. On a single in-process binary this is microseconds. On a multi-service stack (API → queue → worker → browser pool) it's queue wait + inter-service hops + worker pickup, which under load is often the largest hidden segment.
2. DNS + TCP + TLS — resolving the host and the handshake. Largely fixed (~tens of ms with warm caches), but cold connections and proxy chains add round trips. Connection reuse and HTTP/2 matter here.
3. Time to first byte (TTFB) — the target server thinking. Genuinely the site's fault; the scraper can't beat it, only avoid amplifying it.
4. Content transfer — downloading the response. Proportional to page weight; trivial for text, non-trivial for asset-heavy pages if you render.
5. Render decision + render — the big one. Deciding whether the page needs JS execution, and if so, launching/reusing a headless browser. A browser launch is hundreds of ms to seconds. Always rendering is the single biggest self-inflicted latency tax in the category.
6. Extraction — HTML → clean markdown/structured JSON: DOM parse, boilerplate stripping, conversion. In a compiled engine this is sub-millisecond to low-ms; in a heavy interpreted pipeline with chunking it's measurable.

Segments 3 and 4 are the site's; the scraper owns 1, 5, and 6 and influences 2. Most of the controllable latency lives in admission overhead and the render decision.

The render decision is where engines win or lose

The majority of pages on the web do not require JavaScript execution to extract their main content. An engine that launches a headless browser for every request pays a render tax on the ~70%+ of pages that didn't need it. A well-designed engine does a cheap fetch first, detects whether the meaningful content is present in the static HTML, and only escalates to a browser when the page genuinely requires it. This single heuristic is worth more latency than almost any other optimization, because it removes hundreds of ms to seconds from the common path.

The trade-off, stated honestly: an over-eager "skip the browser" heuristic produces empty extractions on JS-heavy SPAs. The engineering is in the detection, not the dogma — fast on the common case, correct escalation on the hard case. Getting this wrong in either direction is why some engines are fast-but-lossy and others are correct-but-slow.

Admission overhead: the cost of a multi-service stack

A scraping stack composed of an API service, a Redis queue, worker processes, and a browser pool has architectural latency that an in-process binary does not: the request is serialized into a queue, a worker polls it, the worker may cold-start a browser, results are written back through the queue. Under low load this adds tens of ms; under contention it adds hundreds and widens the tail dramatically. This is why P95/P99 for distributed scrapers degrades faster than the median when concurrency rises — the queue is a latency amplifier exactly when you're busiest. A single-process engine that handles the request inline has no queue to wait in; its tail stays tighter under load.

Cold start: the segment serverless people forget

If you scale to zero or run ephemeral workers, there's a seventh segment before all others: process/container start to first served request. A multi-service Compose stack must boot an API, connect Redis, warm a worker, and possibly a browser pool — seconds. A single statically-linked binary with a tiny image (~8MB) and minimal runtime starts in the tens-of-ms range. For bursty or serverless scraping, cold start can dominate the experienced latency of the first requests after a scale-up, which is exactly when an agent is most likely waiting.

Why a compiled, single-binary design is structurally faster on the controllable segments

Mapping the segments to architecture:

• Admission: in-process handling → no queue/IPC tax. Tighter tail under load.
• Render decision: fetch-first with escalation → no browser tax on the static majority.
• Extraction: compiled HTML→markdown → sub-ms conversion vs. interpreted-pipeline overhead.
• Cold start: single tiny binary → tens of ms, not seconds.

This is the design fastCRW commits to: one Rust binary, no Redis/queue, browser fallback only on demand. Our public benchmark publishes the full latency distribution with a one-command repro on /benchmarks — but the number is only as good as its reproduction, which is why the benchmark ships as a one-command script. The architectural reason for the lower latency is the segment analysis above, not a magic constant; that's the part you can verify by reasoning, then confirm by running it on your own targets.

The connection-reuse segment people forget to optimize

One latency segment hides between DNS and TTFB and is routinely left on the table: connection establishment amortization. A naive scraper opens a fresh TCP+TLS connection per request — for HTTPS that's multiple round trips of pure handshake before any application data flows, and at scale this dominates the controllable budget more than people expect. An engine with proper connection pooling and HTTP/2 (or HTTP/3) reuse pays the handshake once per host and amortizes it across many requests, collapsing a multi-round-trip cost to near-zero on subsequent fetches of the same origin — which is exactly the access pattern a crawl produces (many pages, same host). This is invisible in a single-URL microbenchmark (one request, one handshake, no reuse to measure) and decisive in a real crawl. It's another reason crawl-shaped benchmarks reveal engine differences that single-page benchmarks hide: connection reuse only pays off across requests, so the metric you choose determines whether you even see it.

Latency variance is a feature of the architecture, not noise

Teams treat latency variance as measurement noise to be averaged away. It is the opposite — variance is a direct readout of the architecture, and it's what an agent actually experiences. A queued multi-service stack has structurally high variance: a request that arrives to an idle worker is fast; the same request arriving when workers are saturated and a browser must cold-start in the pool is many times slower. The distribution, not the mean, is the architecture's signature. An in-process engine with no queue and on-demand render has a tighter distribution because there are fewer states the request can fall into. This is why P99 and the shape of the histogram matter more than the average for agent workloads: an agent making a chain of ten tool calls samples the tail repeatedly, so it experiences something closer to your P99 than your P50 on every conversation. Optimizing the mean while ignoring variance optimizes a number no user ever feels.

How to measure latency honestly

1. Measure single-request wall-clock, not just throughput — they answer different questions.
2. Report P95/P99, not just the mean; the tail is what an agent experiences in a tool chain.
3. Segment your measurement: separate TTFB (site's fault) from admission + render + extraction (scraper's fault) so you optimize the right thing.
4. Measure under your real concurrency; queue-based stacks look fine at concurrency 1 and degrade at concurrency 50.
5. Include cold start if you scale to zero.
6. Demand a reproducible script from any vendor; a latency claim without one is not evidence.

Bottom line

Scraping latency is mostly not the site — it's admission overhead and the render decision, both of which are architecture, not luck. A multi-service queued stack pays a tax on every request and a worse tax under load; a compiled single-binary engine that renders only when needed avoids both and keeps its tail tight. That's why the controllable latency segments favor fastCRW's design, and why the lower-latency framing has a structural explanation you can check rather than a number you have to trust. Measure P95 under your real concurrency, segment site-time from scraper-time, and never accept a speed claim without the repro.

Try it

docker compose up   # in-process, fetch-first, ~6MB, AGPL-3.0

Managed Cloud: one-time lifetime 500 free credits, no card. fastcrw.com · GitHub

Related: Fastest web scraping API · Rust vs Python scrapers · LLM-ready markdown extraction