By the fastCRW team · ArXivQA run completed and scored 2026-06-20 on the deployed endpoints · Verify independently.
The short version
Firecrawl shipped a Research Index: a purpose-built API for research agents — search papers, inspect metadata, read passages, expand the citation graph. On the public ArXivQA paper-retrieval benchmark, their Research Index scores 53.3% recall, the highest of any provider on the board.
We built a drop-in compatible Research API — same endpoint paths, same request/response shapes, so the Firecrawl research SDK works unchanged against api.fastcrw.com — and scored it on the same 191 questions. fastCRW reaches 61.0% recall: +7.7 points ahead of Firecrawl, and clear of Claude (45.4%), Parallel (44.3%), and Exa (43.4%).
The interesting part isn't the number. It's that we got there with no self-hosted paper index at all — every query is answered live. Here's how.
Disclosure: we build fastCRW, a Firecrawl-compatible, open-core (AGPL-3.0) scrape/crawl/map/search engine. This post is about a new endpoint family we shipped. We're honest below about what's live, what's a thin wrapper, and where the recall actually comes from.
The benchmark
ArXivQA is 191 natural-language research questions, each with a set of ground-truth arXiv papers that answer it. You're scored on recall — the fraction of each question's correct papers you surface — averaged over all 191. Extra papers don't hurt, so the whole game is coverage.
The questions are deliberately hard. Many ask for a family of papers ("papers proposing overlong reward shaping", "LoRA variants that insert a new matrix between two existing ones"). Some ask "what does paper X compare against" — where the answer lives inside X's own bibliography. Some ask "which open model is best on benchmark Y" — where you have to read a leaderboard. A title-and-abstract search alone misses most of them.
| Provider | ArXivQA recall |
|---|---|
| fastCRW Research API | 61.0% |
| Firecrawl Research Index | 53.3% |
| Claude | 45.4% |
| Parallel | 44.3% |
| Exa | 43.4% |
No index. Live retrieval.
The obvious way to win a paper-retrieval benchmark is to build a big semantic index over every arXiv paper and search it. We didn't. fastCRW's Research API answers each query live, by merging three retrieval sources at request time:
- Our own search — a self-hosted general web search layer plus a research mode that routes to arXiv, Crossref, and scholarly engines. This is the primary recall driver. The agent rewrites each question into 8–12 exact-name queries — specific method, model, dataset, and benchmark names — and those specific queries surface the niche papers a single broad query never ranks.
- Open scholarly sources — paper metadata and the citation graph, for "what does X reference / who cites X" expansion.
- Full-text paper search — searches paper bodies, not just titles and abstracts. This turned out to matter most: in a source-by-source breakdown it was the single highest-recall source, and it contributed roughly 19% of ground-truth papers that nothing else found — the papers that mention the topic deep in the text but never in the title or abstract.
That last point is the whole reason title-search retrievers cap out. The answer to "papers that do X" is often a paper whose abstract never says "X" — but whose method section does. Searching the body recovers it.
The endpoints are dumb. The skill is smart.
Firecrawl's Research Index is split into stateless endpoints (search-papers, inspect-paper, read-paper, related-papers) plus a research skill that orchestrates them. We mirror that split exactly.
The endpoints — /v1/search/research/papers, /papers/{id}, /papers/{id}/similar, /search/research/github — are stateless primitives. They just retrieve and rank. The intelligence lives in the research skill that drives them:
- Intent routing. "Compare-against" questions → pull the seed paper's own references (the answer is in its bibliography; citation APIs lag months on new papers, the PDF doesn't). "Best on benchmark Y" → read the leaderboard, extract the open model names, search each one's technical report. "Enumerate papers that do X" → exact-name reframings plus, when there's a tight survey, harvest its list.
- Exact-name decomposition, then expansion. One broad query is weak; 8–12 specific-name queries is strong, and that is the single biggest recall lever. The second half of the same pass is what most re-implementations miss: take the five ids the most queries agreed on and pull their references. Measured on the first 12 questions with one raw query each, so the expansion is the only variable, the search union alone scores 47.0% and the expansion takes it to 66.5%.
This is why the headline number is the skill over the endpoints, not the endpoints called blind — the same way Firecrawl's number is measured. Drop-in compatibility means the skill, the SDK, and the CLI all work the same against either provider.
Drop-in compatible
The response shapes match Firecrawl's research SDK field-for-field — paperId, primaryId, prefix-less ids, the two distinct paper shapes for search vs inspect. Point the Firecrawl research SDK at https://api.fastcrw.com and it works:
curl -s -H "Authorization: Bearer $FASTCRW_API_KEY" \
"https://api.fastcrw.com/v1/search/research/papers?query=diffusion%20image%20synthesis&k=20"Or the fastCRW SDK:
from crw import CrwClient
c = CrwClient(api_key="crw_live_…")
c.search_papers("diffusion image synthesis", k=20)
c.related_papers("arxiv:1706.03762", intent="efficient transformers", mode="references")Honest limits
- It's live, so it's seconds, not milliseconds. A hot in-memory index answers in tens of milliseconds; our multi-source live merge takes seconds, and the slowest leg is an upstream source's shared rate limit. We trade a little latency for current coverage and zero index maintenance. For an agent doing deep research, seconds-per-call is fine; for autocomplete, it isn't.
- Read-passages are abstract-scoped today. Full arXiv-body passage retrieval is on the roadmap.
- The number is the agent + skill, run live against the deployed endpoints — all 191 questions, ground truth hidden. It's the product, not a private harness.
Try it
Install the research skill into your agent (Claude Code, Cursor, Codex, Gemini CLI) with one command:
npx skills add us/crw@crw-researchThe Research API docs cover every endpoint, and the ArXivQA benchmark page is the canonical reference for the 61.0%. If you already use Firecrawl's research SDK, the only change is the base URL.