Build a Finance Research Agent in Python

By the fastCRW team · Benchmarks verified 2026-05-18 · scrape: diagnose_3way.py, 2026-05-08; search: triple-bench.ts · Not investment advice · Verify independently before relying on any figure.

What a finance research agent does (and how to build one)

To build a finance research agent you are really building one loop: search for what's happening with a ticker, find the authoritative sources, scrape them to clean text, extract the figures that matter, and hand an LLM a grounded brief to synthesize. The four moving parts are search-find-scrape-analyze, and the order matters — an LLM with no fresh retrieval just hallucinates last quarter's numbers.

The sources a finance agent leans on are predictable: financial news (price action, M&A, analyst moves), investor-relations (IR) pages on the company's own site (press releases, earnings decks, guidance), and regulatory filings (10-K, 10-Q, 8-K). Two properties dominate everything else here: freshness (a stale quote is worse than no quote) and accuracy (a misread revenue line is a wrong conclusion, not a typo).

That is exactly where the retrieval layer earns its keep. fastCRW had the highest truth-recall of the three tools tested — 63.74% of 819 labeled URLs (diagnose_3way.py, Firecrawl's public dataset, 2026-05-08), ahead of Crawl4AI (59.95%) and Firecrawl (56.04%) — and its search averaged 880 ms over a 100-query benchmark with 73 of 100 latency wins versus Firecrawl and Tavily (triple-bench.ts). Cleaner extraction means the agent reasons over complete page content instead of garbled HTML; faster search means the find step doesn't dominate the loop. We'll also be plain about the gaps: there is no screenshot output (a formats: ["screenshot"] request returns HTTP 422), so chart images are out of scope — your agent reads the numbers, not the picture.

Step 1: Find sources with live search

Start by turning a ticker or company name into a list of fresh, authoritative URLs. The /v1/search endpoint runs a live web query and can optionally scrape result content in the same call. Because the API is Firecrawl-compatible, if you already use the Firecrawl SDK you point it at fastCRW with a base-URL swap and the rest of your code is unchanged.

Set a base URL and auth header once, then define a search helper. On self-host the base URL is your own instance (for example http://localhost:3000); managed cloud is https://api.fastcrw.com.

• POST {BASE}/v1/search with body {"query": "NVDA quarterly earnings revenue guidance", "limit": 8} and header Authorization: Bearer $CRW_API_KEY.
• In Python: requests.post(f"{BASE}/v1/search", headers=HEADERS, json={"query": query, "limit": limit}), then read r.json()["data"] for the result list.

Each search query costs 1 credit. Filter the results to domains you trust before you spend any scrape budget — for a finance agent that usually means the company's own IR domain, the SEC, and a short allow-list of established financial press. A simple host filter keeps the agent from analyzing a forum rumor as if it were a filing: parse each result's URL with urllib.parse.urlparse, strip a leading www., and keep only hosts that match (or are a subdomain of) a TRUSTED set such as {"sec.gov", "investor.nvidia.com", "reuters.com", "bloomberg.com"}.

Step 2: Scrape filings and IR pages to clean markdown

Now scrape each trusted URL to LLM-ready markdown with /v1/scrape. The renderer is auto by default and falls back chrome → lightpanda → http, so JavaScript-rendered investor pages (the ones that load financials into a single-page app after first paint) get a real browser only when they need one. Every scrape costs 1 credit regardless of which renderer is used.

• POST {BASE}/v1/scrape with body {"url": url, "formats": ["markdown"]}.
• Read the markdown from r.json()["data"]["markdown"] and keep it alongside the source URL so you can cite it later.

The reason markdown matters for finance specifically: tables and numbers survive the conversion. A revenue-by-segment table or a guidance range comes through as a readable markdown table rather than as a wall of <td> tags or, worse, dropped entirely — which is the practical payoff of that highest-of-three truth-recall. There is no multi-URL batch endpoint, so iterate /v1/scrape concurrently (a thread pool or asyncio) across your source list rather than expecting one call to fan out.

A note on PDF filings

Many primary filings are PDFs, and fastCRW has no document-upload /parse endpoint. For HTML versions of filings (SEC's EDGAR serves HTML alongside PDFs) scrape the HTML page directly. For PDF-only documents, pass them through a dedicated PDF parser and normalize that output to markdown before it reaches the analysis step — see the document-parsing guide. Don't pretend the web scraper does both jobs.

Step 3: Structured extraction of key figures

Free-text markdown is fine for the LLM's narrative reasoning, but for the numbers you want a typed record you can diff over time and store in a database. Use formats: ["json"] with a jsonSchema to pull specific metrics out of a filing or IR page. This is the one operation that adds an LLM token cost on top of its 1-credit scrape (billed as usage-metered LLM credits), and it is single-URL — there is no multi-URL batched /v1/extract, so you call it per document and aggregate yourself.

• Define a JSON Schema for the fields you care about, for example an object with period (string), revenue_usd (number), net_income_usd (number), eps_diluted (number), and guidance_next_q (string), with required: ["period", "revenue_usd"].
• POST {BASE}/v1/scrape with body {"url": url, "formats": ["json"], "jsonSchema": schema}.
• Read the typed record from r.json()["data"]["json"].

Reserve JSON extraction for the pages where structure pays off (earnings releases, the financial-statements section of a 10-Q). For the broad context — management commentary, risk factors, news framing — stick with the 1-credit markdown scrape and let the LLM read prose. Mixing the two keeps cost honest: you are not paying for LLM extraction on a news blurb that has no structure.

Step 4: LLM analysis and a cited brief

With fresh markdown context and a typed metrics record in hand, the final step is synthesis. fastCRW's managed search-answer mode runs a managed LLM for you on paid plans (the FREE tier has no LLM features). For the analysis prompt in a finance agent, where you want a specific reasoning model, the cleanest path is to call your own LLM directly with the scraped evidence — fastCRW hands you clean, grounded context and you own the synthesis model and prompt.

Concatenate each source as a labeled block (the URL as a heading, then the first few thousand characters of its markdown) and pass that as context to your model alongside the structured metrics record. The system instruction does the heavy lifting:

• "Using ONLY the sources below, write a brief on the company's latest quarter."
• "Cite the source URL inline for every figure."
• "If a number is not in the sources, say so — do not estimate."

Then call your own model with that prompt — for example a client.chat.completions.create(...) or messages.create(...) call with whichever reasoning model your stack already uses. The synthesis model is entirely your choice; fastCRW's role ends at handing you clean, grounded evidence.

The non-negotiable part is grounding: instruct the model to cite the source URL for every figure and to refuse to invent numbers that aren't in the retrieved text. That single constraint is what separates a research agent from a confident-sounding guess generator — and pairing live retrieval with an explicit "don't estimate" rule is the most reliable lever for cutting hallucinations, covered in depth in reducing LLM hallucinations with web search.

Honest limits for finance use

A finance agent is exactly the place to be conservative, so here are the limits stated plainly:

• No charts. There is no screenshot output — formats: ["screenshot"] returns HTTP 422. Your agent works from the numbers and tables in the text, never from a rendered chart image. For a stock-price chart, pull the underlying series from a data source, don't try to "read" a picture.
• PDF filings need a separate parser. fastCRW has no document-upload endpoint. Scrape HTML filings directly; route PDF-only documents through a dedicated parser first.
• No batch extract and no built-in state. There is no multi-URL /v1/extract, and the engine is stateless per request — so you iterate scrapes concurrently and persist the time-series of metrics yourself (a Postgres table keyed by ticker and period works well).
• Not investment advice. An agent that summarizes public sources is a research tool. Always trace each figure back to the primary filing before acting on it; treat the brief as a starting point, not a recommendation.

None of these are dealbreakers for a research workflow — they're the difference between a tool you can trust and one that quietly overclaims. If you want to push past summarization into multi-step, autonomous investigation, note that fastCRW deliberately ships the retrieval primitives rather than a managed agent harness; see building a deep research agent for how to orchestrate the loop yourself, and the search API for AI agents for tuning the find step.

Cost, scale, and self-hosting

End to end, a single ticker brief is cheap to reason about: roughly 1 credit per search query, 1 credit per page scraped to markdown (any renderer), and 1 credit plus the usage-metered LLM token cost for each JSON metrics extraction. Run those numbers against your own coverage list and check live tiers on the pricing page rather than trusting a hard-coded table — and remember the AGPL-3.0 engine self-hosts for $0 (you pay only your server), which matters if your research data shouldn't leave your infrastructure. For structured-extraction details, the JSON-schema extraction guide goes deeper on schema design.

Sources

• fastCRW canonical facts : internal fact sheet, verified 2026-05-29.
• Scrape benchmark of record: bench/server-runs/RESULT_3WAY_1000_FULL.md — diagnose_3way.py, Firecrawl public dataset, 819 labeled URLs, 2026-05-08.
• Search benchmark: benchmarks/triple-bench.ts — 100 queries, fastCRW vs Firecrawl vs Tavily.
• SEC EDGAR full-text and filing access: sec.gov/edgar (verify filing formats independently).

Related: Build a deep research agent · Structured JSON-schema extraction · Search API for AI agents · Reduce LLM hallucinations with web search