By the fastCRW team · Comparison post · Catalogue figures checked 2026-11-05 against GET /v1/data · Verify each source's terms independently before relying on it.
Use the official API when the source publishes one and its terms fit what you plan to do. Scrape the page when the data lives only on the page. Most real agents need both, so the useful question is which path to try first for each source, and what to do when the first path fails.
The short answer
An official API gives you named fields, a documented shape and, usually, explicit licence terms. A page gives you what a browser shows, which can include data the API never exposes, but you have to parse it and it changes whenever the layout changes. If a source has an API that covers your question, start there. If it does not, scrape, and keep the source URL and the time you fetched it so the answer can be checked later.
Decision table
| Criterion | Official API | Web scraping |
|---|---|---|
| Structure | Named fields in JSON or XML that you can map to a schema | HTML or rendered markdown that you parse or ask a model to read |
| Stability | A documented schema you can pin your code to | Breaks when the page layout, class names or scripts change |
| Licence | Usually explicit terms and an attribution text in the response | The site's terms of use, which may say little or nothing about automated reads |
| Freshness | As fresh as the source's own update feed | As fresh as the page you fetch; you set the cache policy |
| Coverage | Only what the API exposes | Everything the page shows, including data with no API behind it |
| Cost | One request per answer; on fastCRW a failed data call is not billed | Fetching and rendering; JavaScript-heavy pages take more work to render |
Example 1: a company filing from the SEC
The SEC EDGAR source serves a submissions record for every company, keyed by a ten-digit CIK. Apple's CIK is 0000320193. One call returns the company's identity and its filing history as structured fields.
curl "https://api.fastcrw.com/v1/data/sec/submissions?cik=0000320193" \
-H "Authorization: Bearer $CRW_API_KEY"
Trimmed output from 2026-11-05:
{"success":true,"provider":"sec","capability":"submissions","credits_used":1,
"data":{"cik":"0000320193","name":"Apple Inc.","tickers":["AAPL"],"exchanges":["Nasdaq"],
"sicDescription":"Electronic Computers", ...}}
You could also scrape the SEC's company search page for the same company. It works, but you get markdown that you still have to read:
curl -X POST https://api.fastcrw.com/v1/scrape \
-H "Authorization: Bearer $CRW_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0000320193&type=10-K", "formats": ["markdown"]}'
Trimmed output from the same day begins with "markdown":"# EDGAR Search Results ..., which is the page as text. The company name, the CIK and the filing list are all in there, but you would write the parser yourself, and it would break the next time the page changes. The API route is the right default here.
Example 2: a news article with no API
Many news sites publish no API for their articles. For a page like that, scraping is the only path. Send the article URL to /v1/scrape with the markdown format, as in the call above, and keep three things with the result: the URL, the time you fetched it and the headline. An agent that cites the article later can then point to the exact page it read. Do not scrape pages whose owners have asked crawlers to stay away, and check the site's terms before you store the text.
Using both in one agent
Put the order into the agent, not into the prompt. For each question, the agent first looks for a structured source with the free find call:
curl "https://api.fastcrw.com/v1/data/find?query=company%20filings&limit=5" \
-H "Authorization: Bearer $CRW_API_KEY"
On 2026-11-05 the first item was sec/submissions. If a capability fits, the agent calls it. If none fits, it searches for candidate pages and scrapes only the URLs that the first two steps cannot answer. The output says which path produced each figure, so the structured answers stay the default and the scraped answers are easy to audit.
Reference text is a good fit for the API path too. This is the Wikipedia extract capability, one call, with the response data shown as returned:
curl "https://api.fastcrw.com/v1/data/wikipedia/extract?titles=Web_scraping&exintro=1&exsentences=2" \
-H "Authorization: Bearer $CRW_API_KEY"
{"success":true,"provider":"wikipedia","capability":"extract","credits_used":1,
"data":{"batchcomplete":true,"query":{"normalized":[{"fromencoded":false,"from":"Web_scraping","to":"Web scraping"}],
"pages":[{"pageid":2696619,"ns":0,"title":"Web scraping","extract":"Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. Web scraping software may directly access the World Wide Web using the Hypertext Transfer Protocol (HTTP) or a web browser."}]}}, ...}
The agent cites the extract, the page title and the fetch time, and scrapes only when the question moves beyond what the API returned.
Freshness in practice
Freshness depends on who controls the update. An API inherits the source's own update schedule, so the question is how quickly the source publishes a change. A page is as fresh as the copy you fetched last, and you decide how often to fetch it. For a page, store the fetch time next to every extracted value, so a reader can judge the age of the answer.
Caching adds a layer on the API side too. A REST response from fastCRW reports retrieved_at, cache_age_seconds and is_stale. In our check on 2026-11-05, an SEC submissions record came back with cache_age_seconds at 435 and is_stale set to true. That flag means the body came from cache. It does not mean the SEC had changed the record. If a figure has to be current, call again and check the flag.
Rate limits and etiquette
Official sources publish their limits, and you should read them before you write any loop. The SEC's access page states a maximum request rate of 10 requests per second and asks clients to declare a user agent in the request headers. NCBI's E-utilities guidance asks users to post no more than three URL requests per second, and warns that a client that ignores the policy can be blocked. Both pages are linked in the sources below.
A page has no published limit, but the server behind it still has capacity. Cap your concurrency per host, honour the site's robots rules, and back off on errors instead of retrying in a tight loop. A scrape that hammers a small site is the fastest way to lose the ability to scrape it at all.
Failure modes
The two paths fail differently, so plan for each. An API call can fail upstream. On fastCRW that comes back as success: false with an error code such as upstream_error or upstream_timeout, and a failed data call is not billed. A scrape can succeed while the content is wrong: the page blocked the read, needed a script to render, or changed layout so your parser finds nothing. Validate scraped output against what you expect, and record which path produced each value so a wrong figure can be traced.
A checklist before you scrape
- Search for an official API or feed for the source, and read its terms.
- If one exists and covers the question, use it and stop there.
- If its terms do not fit your use, read the site's terms before you scrape.
- Set a rate limit per host, and respect robots rules.
- Store the URL, the fetch time and the path you used with every value.
What developers say about it
Developers raise this trade-off in public too: a Hacker News post on the subject is titled "Stop crawling my HTML".
Honest limits
- An API can be incomplete or wrong. Check the upstream URL in the provenance before you trust a figure.
- Scraping can break without warning. Keep the URL and the fetch time so a broken result is easy to spot.
- Neither path removes the need to read the source's terms. An API with terms that forbid your use is still a no.
Sources
- SEC: Accessing EDGAR data
- NCBI: E-utilities documentation
- Wikipedia: Web scraping
- fastCRW data sources documentation
Related: curl web scraping guide · Website to JSON extraction · Search API for AI agents · Public data API for AI agents · Pergamon catalogue · Get an API key
