Why does tail latency matter more than average latency for AI agents?

A synchronous scraper call blocks the whole agent loop until it returns, so the worst-case latency — not the median — determines whether a turn stalls or times out. p90 means one call in ten lands at or beyond that latency, so an agent making ten scrape calls per question will hit its p90 about once per question. The average smears that tail into a friendly-looking number and cannot tell you whether your call will time out, which is why you should plan against p90/p99 instead.

What is fastCRW's p90 latency in fast mode?

In fast mode, fastCRW's p90 is 4348 ms — the lowest of the three tools tested (Crawl4AI 4754 ms, Firecrawl 6937 ms; diagnose_3way.py, 2026-05-08). Most URLs resolve on the fast path (http or lightpanda renderer), which is where these numbers come from. For the minority of URLs requiring chrome-stealth recovery — the same path that gives fastCRW 34 exclusive recoveries no competitor reaches — latency rises on those specific pages.

How does truth-recall relate to fastCRW's latency profile?

The chrome-stealth recovery path that gives fastCRW the highest truth-recall — 63.74% of 819 labeled URLs (diagnose_3way.py, 2026-05-08) — and 34 exclusive URL recoveries that no competitor reaches, does add latency on those specific difficult pages. The fast path that handles the large majority of URLs is the fastest by both p50 (1914 ms) and p90 (4348 ms, lowest of three). Both modes are part of the same engine; your URL mix determines which path dominates.

How do I design an agent loop for reliable scrape latency?

Three patterns. First, set an explicit per-call deadline sized against your observed p90 and treat a timeout as a normal branch — proceed with a partial answer rather than hanging the turn; because fastCRW is stateless per request there is no session to clean up. Second, use async /v1/crawl jobs instead of looping synchronous /v1/scrape so per-page latency never touches the reasoning loop. Third, fire multiple scrapes concurrently so your wall-clock wait is the slowest of N, not the sum.

When can I ignore p90 latency entirely?

When the scrape latency never touches a latency-sensitive path. Offline batch or overnight pipelines care about throughput and completeness — per-page latency is noise against a multi-hour run. Cache-then-serve architectures — scrape on a schedule, embed, then serve retrieval from your own index — pay the tail once during ingestion, so the live user-facing request sees only millisecond-scale vector reads. In both cases tail latency is a non-issue by construction.

Honest Tail Latency: When p90 Matters for Agents

By the fastCRW team · Benchmark figures verified 2026-05-18 from a run dated 2026-05-08 (diagnose_3way.py, Firecrawl's public 1,000-URL dataset) · Verify independently before quoting.

Disclosure: We build fastCRW. This post is about what p90 latency actually means for agents — including what our own benchmark shows — because an agent builder who understands the full latency picture is better served than one handed a single flattering average.

Why tail latency, not the median, breaks agents

When you put a web scraper inside an AI agent or a RAG pipeline, the number that hurts you is almost never the median. It is the web scraping tail latency — the p90 and p99 — because that is where a single tool call goes from "fast enough" to "the whole reasoning loop is now hung." The median tells you how a typical request feels in a demo. The tail tells you how your system behaves at three in the morning under real traffic, and the tail is what your users and your timeouts actually experience.

A synchronous tool call blocks the whole loop

An agent that calls a scraper as a synchronous tool does not get to do anything else while that call is outstanding. The model has emitted a tool-use request and is now waiting. If the scrape takes 1.9 seconds, the loop pauses for 1.9 seconds. If it takes significantly longer, the loop pauses accordingly — and any wall-clock deadline on the turn, any user staring at a spinner, any upstream gateway timeout, is now in play. The cost of a slow call is not just that one call; it is the head-of-line blocking it inflicts on everything downstream of it in the reasoning chain.

p90 is one in ten calls, not a rare edge

It is tempting to file the tail under "rare edge case" and move on. Do not. p90 means one request in ten lands at or beyond that latency. An agent that makes ten scrape calls to answer one question will, on average, hit its p90 at least once per question. p99 — one in a hundred — shows up across a day of traffic many times over. Tail latency is not the exception in an agent workload; at any real call volume it is the norm you should plan around.

fastCRW's latency profile (disclosed plainly)

On Firecrawl's own public 1,000-URL scrape-content dataset (diagnose_3way.py, 2026-05-08, single run, 3,000 requests), fastCRW's latency distribution across modes:

Percentile	fastCRW (fast mode)	Crawl4AI	Firecrawl
p50 (median)	1914 ms	1916 ms	2305 ms
p90	4348 ms	4754 ms	6937 ms

In fast mode, fastCRW's p50 of 1914 ms beats Firecrawl's 2305 ms and effectively ties Crawl4AI. In fast mode, fastCRW's p90 of 4348 ms is the lowest of the three — below Crawl4AI's 4754 ms and Firecrawl's 6937 ms. The fast-path median and tail wins are both real, and we publish them together.

The chrome-stealth recovery path and the 34 exclusive URLs

fastCRW's renderer auto-selects with a chrome → lightpanda → http fallback chain. Most URLs resolve on the fast path (reflected in the table above). But when a page needs a full headless browser to render — heavy JavaScript, anti-bot interstitials, late-loading content — fastCRW falls back to a chrome-stealth render rather than returning a thin or empty page. That recovery is exactly what drives the accuracy result: fastCRW returns 63.74% of labeled ground-truth content on the same dataset (522 of 819 labeled URLs, diagnose_3way.py, 2026-05-08), the highest truth-recall of the three tools, ahead of Crawl4AI's 59.95% and Firecrawl's 56.04%. Uniquely, fastCRW recovers 34 URLs that neither Crawl4AI nor Firecrawl reach — 70% more exclusive recoveries than the other two combined. Chrome-stealth recovery adds latency specifically on those difficult pages; on the fast path that covers the large majority of URLs, fastCRW leads on both median and p90.

Why we publish the full split, not a single average

A single "average latency" number cannot tell you whether your synchronous agent call will time out. Always demand the full p50/p90 split — it is the only honest way to describe a distribution. We publish ours at /benchmarks rather than a marketing average.

The accuracy-versus-tail trade-off

Once the cause is on the table, the decision becomes a clean engineering trade rather than a marketing claim.

Highest truth-recall (63.74%) and the lowest fast-mode tail

fastCRW's defining properties on this dataset are recall and fast-path speed: 63.74% of 819 labeled URLs — the best of the three — paired with 91.8% scrape-success of reachable URLs and 0 thrown errors across 3,000 requests, a fast-mode p50 of 1914 ms, and a fast-mode p90 of 4348 ms — the lowest of the three. The chrome-stealth recovery path, which covers the minority of difficult URLs and recovers 34 that no competitor reaches, does add latency on those specific pages. But the fast path that handles the large majority of URLs is the fastest by both median and p90.

When that trade is worth it, and when it is not

For an extraction or RAG pipeline where a missed page means a missing fact in the model's context — and therefore a wrong or hallucinated answer — recovering the hard URLs is usually worth the seconds. A slow-but-complete scrape beats a fast-but-empty one when correctness is the goal. For a latency-bound interactive loop where the user is watching a spinner and a missing page is merely a smaller answer, the calculus flips: you may genuinely prefer a tighter tail and accept lower recall. Both choices are defensible. Neither is served by pretending the tail does not exist.

Designing for latency confidence

For agent and RAG builders, whether the fast-mode path or the chrome-stealth recovery path matters more depends on your URL mix. Either way, the following patterns keep any scraper from stalling your loop.

Per-call timeouts and deadlines

Set an explicit deadline on every scrape tool call and treat the timeout as a normal branch, not an exception. If a call exceeds, say, 8 seconds, cancel it and let the agent proceed with what it has — a partial answer plus a "couldn't retrieve that source in time" note is far better than a hung turn. Because fastCRW is stateless per request, there is no session to clean up after a cancelled call: drop it and move on. Pick the timeout from the percentile data at /benchmarks, not from a hopeful guess — size it against your tolerance for partial results and the URL mix you are actually hitting.

Async crawl jobs instead of synchronous scrape

If you need many pages, do not loop synchronous /v1/scrape calls inside the agent's critical path. Use /v1/crawl, which is asynchronous: it returns a job ID immediately, runs the work off-loop, and lets you poll /v1/crawl/:id for results. The tail latency of any individual page is then hidden entirely from the reasoning loop — the agent kicks off the job, continues thinking, and collects results when they are ready. Moving the slow work off the synchronous path is the single most effective tail-latency mitigation available.

Concurrency to hide tail latency

When you do need several scrapes and want them all before the next reasoning step, fire them concurrently rather than in series. With ten concurrent scrapes, your wall-clock wait is governed by the slowest of the ten, not the sum — so you pay one p90 instead of ten medians stacked end to end. Combine concurrency with per-call deadlines and a "best N of M" rule (proceed once enough sources return), and a single stubborn URL can no longer hold the whole batch hostage.

When p90 does not matter for your workload

Not every workload is latency-sensitive, and for a large class of jobs the tail is simply irrelevant. Be honest with yourself about which camp you are in before you optimize for a problem you do not have.

Offline batch and overnight pipelines

If you are crawling a few thousand pages overnight to refresh a knowledge base, per-page latency is noise against a multi-hour run. Throughput and completeness are what matter there, and fastCRW's high recall, fast-mode p90 of 4348 ms (lowest of the three), and 0-error run are exactly the right properties. Nobody is watching a spinner at 3 a.m.

Cache-then-serve patterns

If your agent reads from a content store that you populate ahead of time — scrape on a schedule, embed, then serve retrieval from your own index — the scrape latency never touches the user-facing request at all. The slow tail is paid once, asynchronously, during ingestion, and your live retrieval path sees only your vector store's millisecond-scale reads. For most production RAG systems this is the right architecture regardless of which scraper you choose, and it makes web scraping tail latency a non-issue by construction.

The honest summary: in fast mode, fastCRW's p90 of 4348 ms is the lowest of the three tools we tested, and fastCRW leads on both truth-recall (63.74%) and exclusive URL recovery (34 URLs no competitor reaches). Chrome-stealth recovery — the path for difficult pages — adds latency on those specific URLs; design around it with deadlines, async crawl, concurrency, and caching, and it stops being a constraint. Publish the full p50/p90 split rather than a single average so stakeholders understand what they are actually getting.

Sources

3-way scrape benchmark of record (truth-recall, scrape-success, p50/p90/p99): diagnose_3way.py against Firecrawl's public scrape-content-dataset-v1 (1,000 URLs, 819 labeled), single run, 3,000 requests, 2026-05-08 — see /benchmarks.
fastCRW renderer fallback (chrome → lightpanda → http), stateless-per-request model, and /v1/crawl async job semantics: github.com/us/crw README.