Results from the AA-Omniscience Benchmark

What this benchmark measures

AA-Omniscience is the answer-accuracy benchmark on the Artificial Analysis Search Index, the public board that scores agent search APIs. It is 600 factual questions spread across six domains: finance, health, humanities and social sciences, law, science engineering and mathematics, and software engineering.

The questions are narrow and the answers are short. A typical one asks which specific clause of an accounting standard governs a named arrangement, or what figure a particular study reported. Median answer length is six characters. The metric is deliberately blunt: LLM-graded accuracy at one attempt, with the same grader applied to every provider on the board.

That shape is what makes it a clean read on retrieval. There is no partial credit and no room for a long answer to hide a miss. Either the search layer put the right document in front of the model or it did not.

Results

fastCRW answered 540 of the 600 questions correctly. Every product currently listed on the board, from every provider, sits below it.

Provider	Accuracy	Gap to fastCRW
fastCRW	90.0%
Firecrawl Search	73	17 points
Exa Search (auto)	70	20 points
You.com Search (highlights)	69	21 points
Exa Search (fast)	69	21 points
Parallel Search (basic)	68	22 points
Parallel Search (fast)	68	22 points
Parallel Search (advanced)	67	23 points
Exa Search (instant)	67	23 points
You.com Search (snippets)	66	24 points
Keenable Search (realtime)	66	24 points
Brave Search	65	25 points
Keenable Search (pro)	65	25 points
Tavily Search (basic)	64	26 points
Parallel Search (turbo)	56	34 points
Model only, no search	38	52 points

Three things are worth pulling out of that table.

The margin is not a rounding difference. The closest product is 17 points behind. The spread across all fourteen listed products is 17 points end to end, from 56 to 73, so the distance between fastCRW and the current leader is as large as the distance between the current leader and last place.

Every provider's best mode is in there. These are not the cheap configurations. Exa auto, You.com highlights and the strongest Parallel tier are the modes those providers put at the top of their own boards, and the pattern holds across all of them.

Search is doing the work, not the model. The same answer model with no search at all scores 38. Every product on the board is measured on how far above that line it lifts the answer. The best listed lift is 35 points. Ours is 52.

Why there is a control run

The board's numbers were produced by Artificial Analysis in their harness. Ours were produced in a rebuild of that harness. Those are two different things, and a score measured in your own harness is worth very little on its own.

So before measuring ourselves we ran a provider that is already listed on the public board through our harness, on the same questions, and compared what we got against what they published.

Run	Our harness	Published on the board
Listed provider, control	74.5%	73

A gap of 1.5 points on a 200-question subset. That bounds how much our rebuild differs from theirs, and it is the only reason we consider our own number publishable. It is a bound, not proof of identity.

On that same 200-question subset, measured side by side in the same harness on the same day:

Run	Accuracy
fastCRW	89.5%
Control provider	74.5%

The configuration that produced it

The default one. A plain request, no flags:

curl -X POST https://fastcrw.com/api/v1/search \
  -H "Authorization: Bearer $CRW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query": "ASC 340-30 deposit arrangement categories", "limit": 10}'

What this number does not say

The grader is a language model. The same grader scores every provider in every run, which makes the comparison internally fair, but it is not a human adjudicator.

The harness is ours. The control run bounds the difference at about a point and a half on this dataset. That is why the number is here, and it is also why the bound is stated rather than buried.

The full write-up, including how the harness was rebuilt and what it cost to run, is in the engineering post.

Reproducing it

Everything above is repeatable. The dataset is AA-Omniscience-Public on Hugging Face, and the harness constants are published in the Artificial Analysis search methodology. The fastCRW side is the single request shown above. The free tier needs no card, so a meaningful subset costs nothing to repeat.

If you rebuild it, run a listed provider as a control before you trust your own number. That one step is the difference between a benchmark and a press release.