In August 2026 Artificial Analysis published a Search Index: a leaderboard that measures search APIs by how much they improve an agent's answers, rather than by how the SERP looks. Fourteen products from seven providers are on it. fastCRW is not one of them.
The obvious move is to email them and ask to be added. We did the other thing first, for a simple reason: a leaderboard entry is permanent, and a number you have not reproduced yourself is not a number you own. So we rebuilt their harness, proved it reads on their scale by reproducing a listed provider's published score, and only then measured ourselves.
This post is the whole run: the methodology, the places we had to deviate and why, the control result, our result, the failure analysis, and the configuration questions we got wrong on the way.
What the benchmark actually measures
The Search Index does not score search results. It scores answers. Every sample runs the same agent loop with the same answer model on the same task, and the only thing that changes between providers is which API powers the agent's web_search tool. The score is how much the search provider lifts the answer quality.
The agent gets three tools:
web_search, which sends the model's query to the search provider and hands back that provider's native response payload, unmodifiedweb_fetch, which pulls one URL from a result and returns the page as textfinish, which submits the final answer
The model gets 25 turns with unlimited tool calls per turn. If it burns all 25 without calling finish, the task scores zero. Every provider is called with max_results=10.
The public Index is the equal-weighted mean of three benchmarks. This post covers AA-Omniscience: 600 factual questions across six domains, and the one of the three we measured on its complete set.
Why we started with AA-Omniscience
Four reasons, in order of how much they mattered.
The dataset is exactly the one they use. ArtificialAnalysis/AA-Omniscience-Public on Hugging Face is 600 rows, six domains at 100 each, which matches the 600 the methodology describes. For the other two legs we would be picking our own subset and introducing a difference we could not measure. Here there is nothing to pick.
The metric is the simplest of the three. LLM-graded accuracy, pass@1. That matters for a rebuild: a benchmark scored against an unpublished rubric would let our grader drift from theirs in a way we could not quantify, and there would be no way to separate a real gain from a grading artefact.
It is the cheapest. Single-hop factual questions resolve in a handful of turns.
It gives us a control we can check. A rebuilt harness is only readable if a listed provider's score on the same leg is public, so we can run them through our harness and compare. Firecrawl Search holds the top AA-Omniscience score at 73, ahead of Exa auto at 70 and You.com highlights at 69, which makes it both the control and the reference point.
The questions themselves are narrow and the answers are tiny. Median question length is 178 characters, median answer length is 6. Roughly 43% of answers are a bare number and most of the rest are a short name or term. Three real examples:
[JavaScript/TypeScript]
Q: In Firefox 89 (64-bit desktop), what is the
maximum allowed ArrayBuffer byteLength for a
raw allocation, as an exact integer in bytes?
A: 8589934592
[Constitutional Law]
Q: Which U.S. Supreme Court Justice wrote the
opinion of the Court in Western Live Stock v.
Bureau of Revenue, 303 U.S. 250 (1938)?
A: Stone
[Economics]
Q: Over what years did Roubini and Sachs examine
15 OECD countries when assessing trends in
tax-to-GDP ratios?
A: 1960-1986
So the thing being measured is precise: can the agent locate one narrow fact that the model does not already know, using the ten results your API returned. The no-search baseline scores 38 on this set, which is the share the answer model simply knows. Everything above 38 is the search provider's contribution.
Rebuilding the harness
We implemented the loop from the published constants: 25 turns, max_results=10, the three tools above, an unsubmitted answer scoring zero, and the same model doing both answering and grading. Artificial Analysis names that model as GPT-5.6 Luna at medium reasoning, which is visible in their own board: every listed entry carries it in its identifier (artificialanalysis.ai/agents/search-api, e.g. firecrawl_serp-gpt-5-6-luna-medium-search-api). We deployed the same model so the comparison holds.
One design decision is worth stating plainly because it is the one that could bias a result. We hold web_fetch identical across every provider, using the same scrape endpoint for all of them. Artificial Analysis publishes provider-specific parameters for web_search only, which implies a shared fetcher on their side too. Varying the fetch tool alongside the search tool would measure a bundle, not a search API, and would quietly hand an advantage to whoever also runs the fetcher. We did not want that advantage, so we removed it.
Three places we had to deviate
Reproducing someone else's harness from a public write-up always turns up gaps. Ours turned up three, and we are listing them because a benchmark you cannot audit is a marketing claim.
| Published | What we did | Why |
|---|---|---|
| temperature 0.6 | omitted | The endpoint rejects the parameter outright for this model. Reasoning models sample internally and do not accept it. |
| endpoint unstated | the Responses API | Chat completions refuses function tools together with a reasoning effort setting. Since Artificial Analysis runs the model at medium reasoning, reasoning is the constraint we kept. |
| grader rubric | our own exact-answer rubric | The rubrics are not published. This is precisely what the control run exists to bound. |
The control run: does our harness read on their scale?
Before measuring ourselves, we ran a provider whose score is already public. Firecrawl Search, configured the way Artificial Analysis configures it (SERP only, no scraping inside the search call), on a stratified 200-question subset:
- Our harness: 74.5%
- Artificial Analysis publishes: 73
Within 1.5 points, on a 200-question subset where the standard error is around 2.3 points. That is the result that makes the rest of this post readable. Our grader is not their grader and our deviations are real, but on the same dataset with the same answer model those differences move the number by about a point and a half, not by ten.
The result
fastCRW, plain search, all 600 AA-Omniscience questions:
accuracy 90.0% (540 / 600)
unfinished 3
turns per task 5.03
searches per task 4.12
fetches per task 2.06
wall time per task 42.4 s
prompt tokens/task 42.0 K
The 200-question subset had given 89.5%, and the full 600 gave 90.0%, so the subset was a fair sample of the whole set.
Against the AA-Omniscience column of the public board:
| Provider | AA-Omniscience accuracy |
|---|---|
| fastCRW | 90.0 (our harness, validated by the control above) |
| Firecrawl Search | 73 |
| Exa Search (auto) | 70 |
| Exa Search (fast) | 69 |
| Parallel Search (basic / fast) | 68 |
| Parallel Search (advanced) | 67 |
| You.com Search | 66 |
| Tavily Search (basic) | 64 |
| no search at all | 38 |
By domain, and it holds across all six:
Software Engineering 97
Humanities & Social 93
Law 90
Finance 88
Science / Eng / Math 87
Health 85
Health and Science are the hardest domains for everyone. On the same subset the control run scored 55.9 on Health and 57.6 on Science, so the gap is widest exactly where the questions are hardest.
What we actually called
This is the part people usually have to guess at, so here it is exactly. Two ordinary requests, nothing bespoke:
POST https://fastcrw.com/api/v1/search
{ "query": "...", "limit": 10 }
POST https://fastcrw.com/api/v1/scrape
{ "url": "...", "formats": ["markdown"], "onlyMainContent": true }
No answer synthesis. No query expansion. No scraping inside the search call. The search response is plain SERP: ten results with url, title, description, snippet and score, handed to the model as-is, because the benchmark requires the provider's native payload to reach the model unmodified.
The scrape call is the shared web_fetch tool, and the agent decides which URL is worth opening. It opened 2.06 pages per task with fastCRW and 3.73 with the control provider, which is its own small signal: when the snippets carry more, the agent has to open fewer pages.
Cost and speed
The Index reports quality alongside cost per 1,000 tasks and time per task, which is the correct way to publish it. Our numbers, measured in our harness:
- Search cost: 4.12 search calls per task, one credit each, one credit is $0.001, so $4.12 per 1,000 tasks, at the public list price any account pays. The cheapest search cost currently published on the board is $8.41.
- Model tokens: 42.0 M input tokens per 1,000 tasks, against 77.7 M for the control provider in the same harness. Fewer turns and fewer page opens means the answer model is the thing you stop paying for.
- Wall time: 42.4 s per task, against 63.5 s for the control provider in our harness.
Cheaper, faster and more accurate is an unusual combination to be able to write down, and it comes from the same place: when the ten results are good enough, the agent stops early.
What the last 10% actually is
We looked at every question that did not score, because the shape of a miss says more than the headline does. Most of them are not retrieval failures at all. In the majority the correct page had already been fetched and the source was in hand. What separates the two figures is that both of them circulate in the published literature, on the founding date of a scientific principle, on a detection rate, on a physical constant, and the benchmark has fixed one of them as the answer. That is a property of the dataset rather than of the search layer, and no amount of better retrieval moves it.
The genuinely search-shaped failures are a smaller group and they share a shape: extremely niche historical, chemical and pharmacological facts where the agent issues 15 to 29 queries and still cannot surface a source. One asked for a 13C NMR chemical shift for allyllithium in THF; the agent issued 22 searches against it and still did not surface a source. Ten of the sixty misses look like that. Failures average 8.02 searches against 4.12 overall, which is worth noting on its own: the agent knows perfectly well when a fact is not surfacing, and keeps working rather than answering confidently from nothing.
What this does not say
We would rather write these limits ourselves than have someone else find them.
This is one leg of three. The Search Index is the mean of three benchmarks, and this post reports AA-Omniscience, measured in full on every question in the set. Generalising 90 to the combined Index would be exactly the sort of claim we are trying not to make.
It is our harness, not theirs. The control run bounds the difference at about a point and a half on this dataset, which is why we consider the number publishable, but it is a bound and not an identity.
The grader is an LLM. Same grader for every provider in every run, which makes the comparison fair, but it is not a human adjudicator.
The result is also published as a standalone benchmark page with the full tables and the control run: AA-Omniscience benchmark results.
Reproducing it
Everything here is repeatable. The dataset is ArtificialAnalysis/AA-Omniscience-Public on Hugging Face. The harness constants are published in the Artificial Analysis search methodology. The fastCRW side is the two requests shown above, and the free tier needs no card, so a meaningful subset costs nothing to repeat.
If you rebuild it, run a listed provider as a control before you trust your own number. That single step is what separates a benchmark from a press release, and in our case it is the only reason we are comfortable publishing a 17-point lead.
What happens next
Every benchmark number we publish carries the control run that makes it readable. A result measured only against itself is not a benchmark, and that standard is the reason this one is worth reading.
If you want to try the configuration that produced this number, it is the default one. A plain POST /v1/search with a query and a limit, no flags, on https://fastcrw.com/api.