What is the difference between a web index and a crawler?

A crawler is the discovery step: a bot follows links and sitemaps, downloads raw HTML, and passes it downstream. A web index is the result of processing that content — parsing, normalizing, chunking, and storing it in inverted and/or vector structures so a query can be answered fast without re-fetching. They run as separate, asynchronous stages: you can crawl without indexing (a one-time scrape) and re-index without re-crawling (re-embedding stored text with a new model).

Why are cached search results sometimes stale?

Because a search result reads from a pre-built index — a snapshot of the web assembled hours, days, or weeks earlier — not from the live page. Between crawls the underlying page can change while the index still reflects the old state. For traditional search slight staleness is tolerable; for AI agents it is dangerous because the model has no signal that its source is outdated and will answer confidently from old data.

Do AI agents need a live crawl or an index?

Most production agents need both. A pre-built index gives cheap, fast, repeated reads and wins on latency and scale; a live crawl gives correctness at the moment of the query and wins on freshness. The common pattern is to serve the bulk of reads from the index and route freshness-critical intents (pricing, availability, compliance, breaking news) to a live scrape or search, then feed those results back to refresh the index.

What is hybrid retrieval?

Hybrid retrieval combines keyword (BM25) and dense vector search over the same index and merges the two ranked lists, usually via Reciprocal Rank Fusion. Keyword search is unbeatable on exact strings like version numbers and product codes; vector search wins on semantic queries where the question's words never appear in the answer. Real workloads contain both, so hybrid dominates in production. Separately, a freshness hybrid layers a fast index against a live crawl for time-sensitive queries.

What Is a Web Index? How It Powers Search & AI Agents

By the fastCRW team · Concepts and benchmark figures verified 2026-05-29 · Verify independently.

What is a web index?

A web index is a pre-built, structured snapshot of web content — a queryable catalog of pages that were crawled, parsed, and stored in advance, so that a query can be answered without fetching any live URLs. When you search Google or ask an AI agent something about the world, neither one is reading the live web at that instant. Both are reading from an index that was assembled hours, days, or weeks earlier. Understanding the web index is the difference between knowing why your agent gives a confident-but-stale answer and being able to fix it.

That distinction matters more for AI engineers than it ever did for SEO. The quality, structure, and freshness of the index your agent reads from set a hard ceiling on what it can accurately answer — no prompt engineering recovers a fact the index never captured, captured wrong, or captured six months ago. This post walks the indexing pipeline end to end, contrasts a pre-built index against a live crawl honestly, and shows where a live scraper like fastCRW fits alongside an index rather than pretending to replace it.

What a web index stores

An index is not a copy of the web. It is a set of derived data structures optimized for one job: fast lookup. Three kinds matter for AI systems:

Keyword (inverted) indexes. Every term maps to a sorted postings list of the documents it appears in, with term frequency and position. Instead of asking "what words are in this document?", you ask "which documents contain this word?" That inversion is what makes lookups fast across billions of pages. Elasticsearch and Lucene are built on it; relevance is scored with BM25.
Vector (dense) indexes. Content is embedded into numeric vectors and stored for approximate nearest-neighbor search, so a query about "affordable laptops" can surface a page that says "budget notebooks." Pinecone, Weaviate, and Qdrant are built around this.
Hybrid indexes. Both at once — keyword for exact precision, vectors for semantic coverage — merged into one ranked list.

Alongside the postings and vectors, a good index records the parts an agent quietly depends on: canonical URLs (so duplicate versions of a page do not split authority), the link graph (for multi-hop reasoning across sources), structured metadata like schema.org types, and — critically for agents — provenance and freshness: where each chunk came from and when it was last seen. A model retrieving a chunk has no native sense of whether it is six hours or six years old. Only the index can tell it.

The four-stage indexing pipeline

Web indexing is a pipeline, and each stage introduces its own failure modes that compound downstream.

Stage	What it does	Where it breaks
1. Crawling	Bots follow links and sitemaps to discover pages and download raw HTML	Crawl budget, robots rules, JS that never renders
2. Parsing & normalization	Strip boilerplate, extract main text, deduplicate, split into chunks	Noise left in, chunk boundaries that split an answer in two
3. Storage & indexation	Store clean content in inverted and/or vector indexes	Wrong index type for the query class
4. Ranking & retrieval	Score what surfaces for a query using keyword, semantic, or hybrid signals	Sparse-vs-dense mismatch; stale scores

Stage 1: crawling

Crawling is discovery. Bots read sitemaps and follow links to find pages, then download raw HTML under constraints — a per-page byte limit, a crawl budget that governs how much of a site gets visited, and robots directives that gate which paths are accessible. JavaScript-heavy pages are the classic blind spot: content behind tabs, accordions, or infinite scroll may never render, so it never enters the index. If a page is not crawled, it does not exist downstream — this coverage gap is the most common reason an agent fails to answer something plainly online. Our deep dive on crawling an entire website from its sitemap covers how to close those gaps.

Stage 2: parsing and normalization

Raw HTML is mostly noise — navigation, ads, footers, cookie banners. Parsing strips that boilerplate and extracts the main text. For keyword search a little leftover noise is tolerable; for AI retrieval it is toxic: a noisy chunk produces a poor embedding, the poor embedding surfaces irrelevant results, and the model writes a confident answer grounded in content that was never relevant. This is also where chunking happens, and bad chunk boundaries are an underrated cause of retrieval collapse — the right answer is in the index, but split mid-thought across two chunks where neither half scores high enough to surface.

This is where most developer time actually goes. Crawling and storage are largely solved infrastructure; clean extraction at scale across diverse page types is the ongoing engineering challenge, and everything downstream is constrained by what extraction produced. It is where fastCRW concentrates: on a 3-way run over Firecrawl's own public 1,000-URL scrape-content-dataset-v1 (diagnose_3way.py, 2026-05-08), fastCRW recorded the highest truth-recall of the three tools tested — 63.74% of the 819 labeled URLs (522/819), versus 59.95% for Crawl4AI and 56.04% for Firecrawl — at ~92% scrape-success (of reachable URLs) with 0 thrown errors. Extraction accuracy feeds embedding quality, so a few points compound into a different retrieval ceiling. See /benchmarks for the full p50/p90 split.

Stage 3: storage and indexation

Parsed content is stored in the structure that fits the queries you will run. The inverted index intersects postings lists for fast, exact keyword lookup. For semantic retrieval, content is embedded and stored for approximate nearest-neighbor search, because comparing a query against every vector does not scale at billions of entries. Production systems usually run both indexes together and merge results with Reciprocal Rank Fusion — keyword search misses synonyms and intent; semantic search misses exact strings like version numbers and product codes; hybrid covers both.

Stage 4: ranking and retrieval

The last stage decides what surfaces for a query, and the signals differ sharply between traditional search and AI retrieval. A search engine layers behavioral signals — clicks, dwell time, link authority — on top of BM25. An AI retrieval system has none of that; it builds relevance from the content itself, which is why extraction quality and chunking discipline (stage 2) dominate the outcome. The tradeoff comes down to query type: keyword retrieval is unbeatable on exact identifiers, dense retrieval wins where the question's words never appear in the answer, and hybrid wins in production because real workloads contain both.

Index vs live crawl: freshness vs latency

This is the trade that decides your architecture, so it is worth stating without spin. A pre-built index and a live crawl optimize for opposite things, and neither is "better" in the abstract.

Dimension	Pre-built index	Live crawl + search
Latency at query time	Near-instant (lookup only)	Higher — you fetch and parse now
Freshness	As old as the last crawl (hours to weeks)	Current as of this second
Scale / cost per query	Amortized — index once, query cheaply many times	Pay the fetch cost on every query
Coverage control	Whatever the crawler reached	Exactly the URLs you point at, on demand
Best for	High-volume, repeated, latency-sensitive reads	Pricing, compliance, news, anything where "stale" means "wrong"

When the index wins: high query volume against a corpus that does not change minute to minute, where sub-second latency matters and you can tolerate a snapshot that is a few hours or days old. The index amortizes the expensive work once and serves cheaply forever after.

When the live crawl wins: anything where staleness is invisible and dangerous. A research agent answering "what are the current requirements for X?" from a six-month-old index returns the old requirements and has no way to know they changed — it answers confidently and wrong. For pricing, compliance, availability, and breaking news, a stale index does not produce a worse result; it produces a wrong one that looks right. The honest framing for any team is that latency and scale push toward the index, while freshness and correctness push toward live retrieval. Our note on scraping latency, explained unpacks where that fetch cost actually goes.

Hybrid retrieval: combining index and live data

The word "hybrid" gets used for two distinct things, and conflating them causes real architecture mistakes.

Retrieval hybrid is keyword plus vector search over the same index — BM25 for precision, dense vectors for coverage, fused into one list. That is the within-index hybrid described above.

Freshness hybrid is the systems-level pattern that matters most for agents: serve the bulk of reads from a cheap, fast pre-built index, and reach out to a live crawl only for the slice of queries where freshness is non-negotiable. A typical layering:

Query the index first. If a fresh-enough, high-confidence chunk exists, use it — fast and cheap.
Route freshness-critical intents (pricing, stock, "latest", "current", today's news) to a live scrape or search instead of trusting the snapshot.
Feed live results back to refresh the index, so the next identical query is cheap again.

fastCRW is built to be the live leg of that pattern. Its Firecrawl-compatible /v1/search returns web results with optional content scraping in one call, and on a 100-query benchmark across 10 categories (triple-bench.ts) fastCRW search averaged 880 ms with 73 of 100 latency wins against Firecrawl and Tavily (search latency only, not scrape accuracy). For the within-index design the live leg complements, see the web context layer for AI agents and choosing a search API for AI agents.

Why index quality caps what your agent can answer

For a search engine the index is one component in a system that returns links. For an AI agent the index is the knowledge base — everything the agent says is downstream of what the index contains and how it is structured. Three failure modes recur in production, and each traces back to a different stage of the pipeline:

Coverage gap (stage 1). The page exists but the crawler never reached it. The agent cannot answer from what was never indexed.
Noise (stage 2). The page was crawled but extraction left boilerplate in. The bad embedding never surfaces in retrieval, or surfaces and misleads.
Bad chunks (stage 2). The answer was split across a chunk boundary; neither half scores high enough, so the correct content never reaches the model.
Staleness (freshness). The index reflects a state of the world that has since changed, and the model has no signal that its source is old.

None of these is a model problem, and prompt engineering fixes none of them. The fix is always earlier in the pipeline — better coverage, cleaner extraction, smarter chunking, fresher data. That is the whole reason "indexed for Google" does not mean "indexed for AI": Google optimizes for click-through on a top-10 list, while an agent needs top-k chunks that are complete, accurate, and self-contained enough to answer without reading around them.

Where a live scraper fits alongside an index

fastCRW does not maintain its own persistent, pre-built web index. It is stateless per request: a single static Rust binary that crawls, maps, scrapes, and searches on demand. That is a deliberate boundary. You bring the index — your vector store, your Elasticsearch cluster, your RAG pipeline — and fastCRW supplies the fresh, low-noise content that fills it and the live retrieval that bypasses it when freshness wins.

Concretely, fastCRW is the right fit for three jobs around an index:

Building and refreshing the index. Crawl a site, extract clean LLM-ready markdown or structured JSON, and write it to your store. Clean extraction at the input directly raises the retrieval ceiling.
Serving the freshness leg. Route "current/latest/today" intents to /v1/search or a live /v1/scrape so the agent never answers a time-sensitive question from a stale snapshot.
Keeping the cost ceiling and the data in your control. The engine is AGPL-3.0 and self-hostable, so heavy recurring index refreshes have a hard worst-case cost — your server bill — and in self-host mode scraped content and target URLs never leave your infrastructure, which matters for regulated corpora.

Because the API is Firecrawl-compatible, wiring fastCRW into an existing pipeline is a base-URL swap in the Firecrawl SDK:

from firecrawl import FirecrawlApp

app = FirecrawlApp(
    api_key="YOUR_FASTCRW_KEY",
    api_url="https://api.fastcrw.com",  # the only change
)

# Live leg: fresh content for the freshness-critical slice
res = app.search("latest pricing changes acme.com", limit=5)

Be clear-eyed about the boundaries too: fastCRW has no batched multi-URL extract, no /v1/agent or deep-research endpoints, and LLM-based extraction runs on fastCRW's managed LLM, on paid plans only. (Screenshots are supported — a request for formats: ["screenshot"] returns data.screenshot as a base64 PNG data URL.) It is a focused live-retrieval and extraction engine, not an index host. For managed-cloud pricing see /pricing (Free starts with 500 one-time credits); self-hosting the engine is free beyond your own server.

The one-sentence takeaway

A web index gives you cheap, fast, repeated reads of a snapshot; a live crawl gives you correctness at the moment of the query — so make the index your default read layer, route the freshness-critical slice to live retrieval, and feed the live results back. Your agent is only as good as its index, and your index is only as good as the extraction that fills it.

Sources

fastCRW scrape benchmark — Firecrawl's public scrape-content-dataset-v1, 819 labeled URLs, diagnose_3way.py, 2026-05-08: truth-recall 63.74% (522/819) — highest, ~92% scrape-success (of reachable URLs), 0 thrown errors, p50 1914 ms (fastest), p90 4348 ms in fast mode (lowest).
fastCRW search benchmark — triple-bench.ts, 100 queries / 10 categories: avg 880 ms, 73/100 latency wins. Search latency only.
fastCRW repo, pricing, and benchmarks: github.com/us/crw · /pricing · /benchmarks