Results from the ArXivQA Research-Retrieval Benchmark

What this benchmark measures

ArXivQA is alphaXiv's public paper-retrieval benchmark: 191 natural-language research questions, each with a set of ground-truth arXiv papers that answer it. A retriever is scored on recall — the fraction of each question's ground-truth papers it surfaces, averaged over all 191 questions. Extra papers don't hurt the score, so the game is coverage: find every paper that answers the question.

The questions are hard on purpose. Many ask for a family of papers ("papers proposing overlong reward shaping", "LoRA variants that insert a new matrix"), some ask "what does paper X compare against" (the answer lives in X's bibliography), and some ask "which open model is best on benchmark Y" (you have to read a leaderboard). A title-and-abstract search alone misses most of them.

Results

Provider	Recall
fastCRW Research API	61.0%
Firecrawl Research Index	53.3%
Claude	45.4%
Parallel	44.3%
Exa	43.4%

fastCRW reaches 61.0% — the highest provider on the board, +7.7 points ahead of Firecrawl's Research Index and well clear of the rest. Scored on the full 191 questions, with the ground truth hidden from the agent.

How it works — live, no index

fastCRW's Research API is not a pre-built paper index. Each query is answered live by merging multiple retrieval sources at request time:

• Our own search (self-hosted) — web + a research mode that routes to arXiv, Crossref, and scholarly engines. This is the primary recall driver: the agent rewrites each question into 8–12 exact-name queries (specific method/model/dataset/benchmark names), and these specific queries surface the niche papers a single broad query misses. It then expands the five ids the most queries agreed on through their references, which is where the papers no query names directly come from. Measured on the first 12 questions with one raw query each, so the expansion is the only variable: 47.0% for the search union alone, 66.5% with the expansion.
• Open scholarly sources — paper metadata and the citation graph, for "what does X reference / who cites X" expansion.
• Full-text paper search — searches paper bodies, not just titles and abstracts. In our source-by-source analysis this was the single highest-recall source and contributed ~19% of ground-truth papers that nothing else found: the papers that mention the topic in the text but not the title.

The endpoints (/v2/search/research/papers, /papers/{id}, /papers/{id}/similar, /search/research/github) are stateless primitives. The intelligence (the two-step base pass above, intent routing, reading a leaderboard, pulling a paper's self-references for "compare-against" questions) lives in the open research skill, exactly the way Firecrawl splits its Research Index endpoints from its research skill. The harness that produced this number, its scorer and the stored 191-question run are public in bench/arxivqa.

Honest scope

• The score is the agent + skill over the live endpoints, the same shape Firecrawl's number is measured in. It is not the endpoints called blind.
• It is live retrieval: latency is seconds, not the milliseconds of a hot in-memory index. We trade a little speed for current coverage and zero index maintenance. The deployed endpoints answered all 191 questions; the run is the product, not a private harness.
• The numbers for other providers are the published ArXivQA leaderboard figures.

Reproduce it

The endpoints are a drop-in target for the Firecrawl research SDK — point it at https://api.fastcrw.com and the /v2/search/research/* surface matches. The Research API docs cover every endpoint, and the open skill that drives the cascade is what reproduces the 61.0%.