By the fastCRW team · Benchmark figures verified 2026-05-18 · Verify independently before quoting them internally.
Search index vs live web, defined
When you build an AI agent's retrieval stack, you face one architectural fork early: do you query a search index — a pre-built, cached snapshot of the web — or do you hit the live web at query time? The honest answer for most production agents is "both," but the two layers solve different problems, fail in different ways, and cost different amounts of latency. This post draws the line between them and shows how to compose them so an agent stays both fast and fresh.
What an index stores and caches
A search index is a data structure built ahead of time. A crawler visits pages on some schedule, parses them, and writes an inverted index (term → documents) or a set of vector embeddings (semantic neighbors), plus a cached copy or summary of each page's content. At query time you are searching that snapshot, not the page itself. Google, Bing, and embedding-based stores all work this way: the expensive crawl-and-parse work happened minutes, hours, or days before your query, so the query itself is cheap. The trade is that you are always reading a copy whose freshness is bounded by the last crawl.
What live retrieval fetches at query time
Live retrieval inverts the timing. Instead of reading a cached snapshot, the agent issues a query, gets a result set, and then fetches the underlying pages right now — parsing them on the spot into clean, model-ready text. Nothing is pre-stored, so there is no staleness window: you see what the page says at the moment of the query. The cost moves from "cheap query against a warm index" to "a network fetch and parse per result," which is exactly the latency you are trying to manage in an agent loop.
Strengths and limits of an index
Low latency and scale
An index's superpower is amortized cost. Because the heavy work is done in advance, a query against a mature index returns in milliseconds and scales to billions of documents. If your agent answers questions about stable, slow-changing knowledge — API references, encyclopedic facts, documentation that updates monthly — an index is the right default. You are not paying a live fetch tax on every turn for content that has not changed since the last crawl.
Staleness and coverage gaps
The two failure modes of an index are staleness and coverage. Staleness: the index reflects the last crawl, so prices, stock levels, scores, and breaking news can be wrong by exactly the crawl interval — and the agent has no way to know the cached value is stale. Coverage: no index crawls everything. Long-tail pages, freshly published content, paywalled or login-gated material, and sites the crawler deprioritized simply are not there. When an agent confidently answers from a stale or missing entry, that is where hallucination-adjacent errors creep in — the model is faithful to its retrieved context, but the context was wrong. See what a web index actually is for how these snapshots are built and why they drift.
When agents must hit the live web
Breaking news and real-time signals
Any time the answer changes faster than your index refreshes, the index is the wrong tool. Current prices, "is this in stock right now," live sports and election results, today's exchange rate, the status of an outage, the latest version of a fast-moving docs page — these demand a live fetch. A snapshot taken even an hour ago can be confidently, precisely wrong. For these queries the agent has to read the page as it exists at query time, which is the whole job of a live-web layer.
Long-tail pages not in any index
The second case is coverage, not freshness. If the page your agent needs was published five minutes ago, lives deep in a site's long tail, or sits behind a parameterized URL no general crawler bothered to index, then no amount of waiting for a refresh will surface it — it is not in the index at all. Live retrieval reaches these pages directly because it does not depend on a prior crawl having found them. This is the difference between "search a cached map of the web" and "go fetch this specific corner of it now." Agentic patterns lean on this constantly; agentic search explained covers how an agent decides to reach for fresh retrieval mid-reasoning.
Combining both: hybrid retrieval
The production answer is rarely "index only" or "live only." It is a hybrid where the index handles the cheap, stable majority of queries and the live web handles the fresh, long-tail minority. Designing the handoff between them is the real engineering work.
Index first, live fallback
The most common and most economical pattern is index-first with a live fallback. The agent queries the index first; if it gets a confident, fresh-enough hit, it answers from the cache and pays no live-fetch tax. If the index misses, returns low-confidence results, or the query is flagged as freshness-sensitive (prices, news, "current," "today," "latest"), the agent escalates to a live fetch. This keeps the common case fast while guaranteeing the agent can always reach ground truth when it matters. The decision rule — "is this query freshness-sensitive or a likely coverage gap?" — is worth encoding explicitly rather than escalating on every turn, because every live fetch costs latency.
Freshness checks and cache invalidation
If you maintain your own index or content cache, the index-first pattern needs a freshness policy. Stamp every cached entry with a fetch timestamp and define a per-domain or per-content-type TTL: a docs page might be fresh for a week, a product price for minutes, a news article for seconds. When a query targets content past its TTL, treat the cache as a miss and fall through to a live fetch, then write the fresh result back. Cache invalidation is famously hard, but for retrieval you can sidestep most of it by being conservative: when in doubt about freshness, fetch live and update the cache as a side effect.
Building the live-web layer
This is where fastCRW fits. Disclosure: we build fastCRW, so weigh the framing accordingly — but the architectural point stands regardless of vendor. fastCRW is a live-web layer, not an index. It maintains no proprietary web index and stores nothing between requests — it is stateless per request (CANONICAL-FACTS §1, §9), so every /v1/search call queries fresh and every result you scrape is the page as it exists now. That is exactly the role the live half of a hybrid stack needs filled: it complements an index, it does not replace one.
Search plus per-result content
A live-web layer for agents needs two things in one round trip: a result set and the actual page content behind each result. fastCRW's /v1/search does both — it runs the query and can optionally scrape the content of each result in the same call (CANONICAL-FACTS §6), so the agent receives clean, model-ready text rather than just a list of URLs it then has to fetch separately. That collapses the classic "search, then fetch each link, then parse" sequence into a single request, which is the difference between one network round trip and a dozen.
Latency budgets in an agent loop
The fear with any live-web layer is that fetching at query time blows the agent's latency budget. The benchmark says otherwise, and the honest framing is to cite the raw numbers, not a speed multiple. On a 100-query search benchmark (benchmarks/triple-bench.ts, run concurrently against all three providers, verified 2026-05-18), fastCRW search averaged 880 ms with a median of 785 ms and a P95 of 1,433 ms, taking 73 of 100 latency wins against Firecrawl and Tavily. A sub-second median with a P95 inside a second and a half is comfortably within an agent turn's budget — live retrieval does not have to mean a slow agent. See the public benchmarks for the full per-provider split, and measure on your own query mix before quoting these internally.
One caveat we state plainly: this is the search benchmark, not the scrape benchmark. If your live layer also deep-scrapes heavy, JavaScript-rendered, or anti-bot-protected pages, the tail latency of that scrape is a separate cost with its own distribution — fastCRW's scrape p90 is the worst of the three tools we benchmark, a deliberate trade for the highest truth-recall. Budget for the work you are actually doing, not just the search leg. For the broader picture of where this layer sits in an agent's stack, see the web context layer for AI agents.
The decision, in one line
Use an index for the cheap, stable majority of queries; use the live web for anything freshness-sensitive or off the indexed path; and wire them as index-first with a live fallback so the common case stays fast and the agent can always reach ground truth. fastCRW is built to be the live half of that pair — stateless, fresh, search-plus-content in one call — not the index half. Knowing which layer a given query belongs to is the design decision that keeps an agent both fast and correct. If you are choosing a search backend for the live layer specifically, the search API for AI agents guide compares the options.
Sources
- fastCRW canonical fact sheet — search benchmark (§6,
benchmarks/triple-bench.ts, 100 queries) and stateless-per-request design (§1, §9), verified 2026-05-18. - fastCRW search and per-result content scraping:
/v1/search(CANONICAL-FACTS §6) · github.com/us/crw · fastcrw.com
Related: What is a web index · Agentic search explained · The web context layer for AI agents · Search API for AI agents