Skip to main content
Comparison

Web Scraping vs the Official API: When to Use Which

When to scrape a page and when to call the source's API: freshness, licence, structure, rate limits and cost, with a decision table and examples.

fastcrw
By RecepNovember 20, 20266 min read

By the fastCRW team · Comparison post · Catalogue figures checked 2026-11-05 against GET /v1/data · Verify each source's terms independently before relying on it.

Use the official API when the source publishes one and its terms fit what you plan to do. Scrape the page when the data lives only on the page. Most real agents need both, so the useful question is which path to try first for each source, and what to do when the first path fails.

The short answer

An official API gives you named fields, a documented shape and, usually, explicit licence terms. A page gives you what a browser shows, which can include data the API never exposes, but you have to parse it and it changes whenever the layout changes. If a source has an API that covers your question, start there. If it does not, scrape, and keep the source URL and the time you fetched it so the answer can be checked later.

Decision table

CriterionOfficial APIWeb scraping
StructureNamed fields in JSON or XML that you can map to a schemaHTML or rendered markdown that you parse or ask a model to read
StabilityA documented schema you can pin your code toBreaks when the page layout, class names or scripts change
LicenceUsually explicit terms and an attribution text in the responseThe site's terms of use, which may say little or nothing about automated reads
FreshnessAs fresh as the source's own update feedAs fresh as the page you fetch; you set the cache policy
CoverageOnly what the API exposesEverything the page shows, including data with no API behind it
CostOne request per answer; on fastCRW a failed data call is not billedFetching and rendering; JavaScript-heavy pages take more work to render

Example 1: a company filing from the SEC

The SEC EDGAR source serves a submissions record for every company, keyed by a ten-digit CIK. Apple's CIK is 0000320193. One call returns the company's identity and its filing history as structured fields.

curl "https://api.fastcrw.com/v1/data/sec/submissions?cik=0000320193" \
  -H "Authorization: Bearer $CRW_API_KEY"

Trimmed output from 2026-11-05:

{"success":true,"provider":"sec","capability":"submissions","credits_used":1,
 "data":{"cik":"0000320193","name":"Apple Inc.","tickers":["AAPL"],"exchanges":["Nasdaq"],
 "sicDescription":"Electronic Computers", ...}}

You could also scrape the SEC's company search page for the same company. It works, but you get markdown that you still have to read:

curl -X POST https://api.fastcrw.com/v1/scrape \
  -H "Authorization: Bearer $CRW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.sec.gov/cgi-bin/browse-edgar?action=getcompany&CIK=0000320193&type=10-K", "formats": ["markdown"]}'

Trimmed output from the same day begins with "markdown":"# EDGAR Search Results ..., which is the page as text. The company name, the CIK and the filing list are all in there, but you would write the parser yourself, and it would break the next time the page changes. The API route is the right default here.

Example 2: a news article with no API

Many news sites publish no API for their articles. For a page like that, scraping is the only path. Send the article URL to /v1/scrape with the markdown format, as in the call above, and keep three things with the result: the URL, the time you fetched it and the headline. An agent that cites the article later can then point to the exact page it read. Do not scrape pages whose owners have asked crawlers to stay away, and check the site's terms before you store the text.

Using both in one agent

Put the order into the agent, not into the prompt. For each question, the agent first looks for a structured source with the free find call:

curl "https://api.fastcrw.com/v1/data/find?query=company%20filings&limit=5" \
  -H "Authorization: Bearer $CRW_API_KEY"

On 2026-11-05 the first item was sec/submissions. If a capability fits, the agent calls it. If none fits, it searches for candidate pages and scrapes only the URLs that the first two steps cannot answer. The output says which path produced each figure, so the structured answers stay the default and the scraped answers are easy to audit.

Reference text is a good fit for the API path too. This is the Wikipedia extract capability, one call, with the response data shown as returned:

curl "https://api.fastcrw.com/v1/data/wikipedia/extract?titles=Web_scraping&exintro=1&exsentences=2" \
  -H "Authorization: Bearer $CRW_API_KEY"
{"success":true,"provider":"wikipedia","capability":"extract","credits_used":1,
 "data":{"batchcomplete":true,"query":{"normalized":[{"fromencoded":false,"from":"Web_scraping","to":"Web scraping"}],
 "pages":[{"pageid":2696619,"ns":0,"title":"Web scraping","extract":"Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. Web scraping software may directly access the World Wide Web using the Hypertext Transfer Protocol (HTTP) or a web browser."}]}}, ...}

The agent cites the extract, the page title and the fetch time, and scrapes only when the question moves beyond what the API returned.

Freshness in practice

Freshness depends on who controls the update. An API inherits the source's own update schedule, so the question is how quickly the source publishes a change. A page is as fresh as the copy you fetched last, and you decide how often to fetch it. For a page, store the fetch time next to every extracted value, so a reader can judge the age of the answer.

Caching adds a layer on the API side too. A REST response from fastCRW reports retrieved_at, cache_age_seconds and is_stale. In our check on 2026-11-05, an SEC submissions record came back with cache_age_seconds at 435 and is_stale set to true. That flag means the body came from cache. It does not mean the SEC had changed the record. If a figure has to be current, call again and check the flag.

Rate limits and etiquette

Official sources publish their limits, and you should read them before you write any loop. The SEC's access page states a maximum request rate of 10 requests per second and asks clients to declare a user agent in the request headers. NCBI's E-utilities guidance asks users to post no more than three URL requests per second, and warns that a client that ignores the policy can be blocked. Both pages are linked in the sources below.

A page has no published limit, but the server behind it still has capacity. Cap your concurrency per host, honour the site's robots rules, and back off on errors instead of retrying in a tight loop. A scrape that hammers a small site is the fastest way to lose the ability to scrape it at all.

Failure modes

The two paths fail differently, so plan for each. An API call can fail upstream. On fastCRW that comes back as success: false with an error code such as upstream_error or upstream_timeout, and a failed data call is not billed. A scrape can succeed while the content is wrong: the page blocked the read, needed a script to render, or changed layout so your parser finds nothing. Validate scraped output against what you expect, and record which path produced each value so a wrong figure can be traced.

A checklist before you scrape

  1. Search for an official API or feed for the source, and read its terms.
  2. If one exists and covers the question, use it and stop there.
  3. If its terms do not fit your use, read the site's terms before you scrape.
  4. Set a rate limit per host, and respect robots rules.
  5. Store the URL, the fetch time and the path you used with every value.

What developers say about it

Developers raise this trade-off in public too: a Hacker News post on the subject is titled "Stop crawling my HTML".

Honest limits

  • An API can be incomplete or wrong. Check the upstream URL in the provenance before you trust a figure.
  • Scraping can break without warning. Keep the URL and the fetch time so a broken result is easy to spot.
  • Neither path removes the need to read the source's terms. An API with terms that forbid your use is still a no.

Sources

Related: curl web scraping guide · Website to JSON extraction · Search API for AI agents · Public data API for AI agents · Pergamon catalogue · Get an API key

FAQ

Frequently asked questions

Is scraping a site legal?
That depends on the site, the jurisdiction and what you do with the data. Read the site's terms of use and any robots rules, and check whether an API covers your need. This post explains the technical trade-off and does not give legal advice.
Should an agent try the API or the scrape first?
Try the API first when one covers the question and its terms fit. It returns named fields that you can validate, and a failed call on fastCRW is not billed. Fall back to scraping only when the data lives on the page.
How do I know a scraped answer is still correct?
Store the source URL and the fetch time with every answer. Re-fetch the page when the answer matters. A scraped result has no schema to validate, so the check is a second read of the same page.

Get Started

Try fastCRW free

Run a live request in the playground — no signup required. Or grab a free API key with 1000 free credits, no credit card.

Continue exploring

More comparison posts

View category archive