By the fastCRW team · Catalogue figures checked 2026-11-05 against GET /v1/data · Verify independently before relying on any figure.
A public data API for AI agents is one interface that returns official public records (SEC filings, CVE records, research papers, national statistics) as JSON, with the licence and attribution of each source attached to the answer. Pergamon, the public data catalogue on fastCRW, gives an agent more than 430 sources behind one API key and one credit wallet. Each source is still its official API underneath. Pergamon makes the call, returns the upstream data, and tells you where it came from.
This post explains why an agent needs official data instead of a search snippet, what the calls look like, what they cost, and where Pergamon does not fit. Every call shown here was run on 2026-11-05 and the output is trimmed, not edited.
Why agents need official data, not a search snippet
A search snippet tells an agent what a page says. It does not tell the agent which filing, which dataset row or which vulnerability record a number came from, or whether the publisher allows reuse. An agent that cannot show its source will still answer with confidence.
Developers have made the same point in public. A Hacker News thread on a post titled "Stop crawling my HTML" (news.ycombinator.com) argues for calling a publisher's API instead of parsing its pages. Another is titled "Twice this week, I have come across embarassingly bad data" (news.ycombinator.com). If a number looks wrong, a person should be able to open the upstream URL the agent used and check it.
The integration tax: keys, limits, shapes, licences
One public API is easy to call. Twenty of them is a maintenance job. The pain shows up in the same four places every time.
- Keys and billing. A r/LangChain thread describes the problem: "You end up holding keys for five different providers, each with their own rate limits, billing cycles, and quota systems..." (reddit.com).
- Rate limits that differ by agency. The SEC publishes "Current max request rate: 10 requests/second" and asks clients to "declare your user agent in request headers" (sec.gov). NVD allows 5 requests in a rolling 30 second window without a key and 50 with one (nvd.nist.gov). NCBI "recommends that users post no more than three URL requests per second" (ncbi.nlm.nih.gov).
- Licences that are hard to find. A Companies House forum user asked in 2023 "What are the rules regarding commercial use of the API?" (forum.companieshouse.gov.uk). MusicBrainz states that "Non-commercial use of this web service is free" (musicbrainz.org).
- Shapes that do not match. Each source returns its own JSON layout, its own error format and its own paging. Your agent needs a parser per source, and a fix per source when the shape changes.
How Pergamon works: discover, call, attribute
Pergamon is built around three steps and one key. The agent finds a source, calls one capability, and reads the provenance block that comes back with the data. Find and catalogue calls are free. A server source call costs the creditsCost of its capability, which is 1 for every server capability used in this post.
Find a source (free) with /v1/data/find
Describe what you need in plain words. The find call is a GET request, it never bills, and limit accepts values from 1 to 50.
curl "https://api.fastcrw.com/v1/data/find?query=company%20filings&limit=5" \
-H "Authorization: Bearer $CRW_API_KEY"The response (trimmed to the first item) tells you the provider, the capability, the cost, the options and the licence:
{
"total": 575,
"items": [
{
"id": "sec/submissions",
"provider": "sec",
"capability": "submissions",
"name": "Company filing history",
"category": "Finance",
"execution": "server",
"creditsCost": 1,
"keyRequired": false,
"options": { "required": ["cik"], "optional": [] },
"provenance": { "licence": "A work of the United States Government, not subject to domestic copyright under 17 U.S.C. 105, published by the agency as open data." }
}
]
}Call one capability
Take the provider and capability from the find result and pass its options. The example below is the SEC filing history for Apple, CIK 0000320193. The CIK must be the 10-digit, zero-padded form. With cik=320193 the upstream answers 404.
curl "https://api.fastcrw.com/v1/data/sec/submissions?cik=0000320193" \
-H "Authorization: Bearer $CRW_API_KEY"Trimmed response:
{
"success": true,
"provider": "sec",
"capability": "submissions",
"credits_used": 1,
"provenance": {
"source": "SEC EDGAR",
"upstream_url": "https://data.sec.gov/submissions/CIK0000320193.json",
"...": "..."
},
"data": { "name": "Apple Inc.", "...": "..." }
}The Python SDK runs the same call. It reads CRW_API_KEY from the environment and defaults to the same API base. This example calls NVD for one CVE:
# pip install crw
from crw import CrwClient
client = CrwClient() # reads CRW_API_KEY
result = client.get_data("nvd", "cve", {"cveId": "CVE-2021-44228"})
print(result["provenance"]["attributionText"])
data = result["data"]The printed line is Source: NIST National Vulnerability Database. The SDK returns data exactly as the upstream sent it, so the field names are the source's own. Errors raise an exception that carries an error_code, for example invalid_options when a required parameter is missing, so your code can branch on the code and not on the message text.
REST and the SDK return the whole upstream body. There is no 256 KiB cap on those surfaces, so a large file comes back in full and your code has to be ready to hold it. The MCP surfaces do cap results, and that difference is covered in the MCP companion article.
A time series works the same way. World Bank GDP for Germany, the last five years:
curl "https://api.fastcrw.com/v1/data/worldbank/indicator?country=DE&indicator=NY.GDP.MKTP.CD&mrv=5" \
-H "Authorization: Bearer $CRW_API_KEY"The World Bank returns a two-element array: paging metadata first, then one object per year. The rows carry date, value, country and indicator, and the response keeps the bank's own names for each.
Read the provenance block
Every REST response carries a provenance block. For the SEC call above it reads:
"provenance": {
"source": "SEC EDGAR",
"upstream_url": "https://data.sec.gov/submissions/CIK0000320193.json",
"licence": "A work of the United States Government, not subject to domestic copyright under 17 U.S.C. 105, published by the agency as open data.",
"licence_url": "https://www.sec.gov/about/privacy-information#security",
"attribution_text": "Source: U.S. Securities and Exchange Commission, EDGAR",
"retrieved_at": "2026-11-05T10:19:52.312Z",
"adapter": "rest",
"cache_age_seconds": 0,
"is_stale": false
}provenance.is_stale is true whenever the body was served from fastCRW's response cache, and cache_age_seconds says how old that copy is. It does not mean the upstream data is out of date. The MCP tool and the Python SDK return the same facts in camelCase, such as attributionText and retrievedAt, without the two cache fields.
Put the attribution_text next to any figure you show a user. That one line is what lets a reader see that the number came from the SEC, the World Bank or NIST, and not from the agent.
What is in the catalogue
The catalogue lists more than 430 sources and 4,200 capabilities in 13 categories. More than 70 of the sources run on your machine and the rest run on fastCRW. These were the seven largest categories on 2026-11-05:
| Category | Sources | Capabilities |
|---|---|---|
| Open data portals | 273 | 2,057 |
| Research | 25 | 340 |
| Developer | 23 | 210 |
| Finance | 17 | 359 |
| Reference | 15 | 179 |
| Health | 15 | 149 |
| Geo | 14 | 261 |
Open data portals is the deepest category. Most of its sources are city, state and national portals. Counts change as sources are added or switched off, so check the live catalogue before you quote one.
To look inside one category, pass category to the find call. The match is case-insensitive, so Finance and finance return the same results. Find is free, so narrowing the search costs nothing. The same call with a provider filter shows the capabilities of one source, which is the fastest way to learn a source's options before you write any code.
The difference from writing one client per API is mostly in the code you do not have to keep. Your agent gets one key, one error shape and one provenance block. A new source is a new provider value, not a new dependency, a new auth flow and a new parser.
Sources that run on your machine
More than 70 of the sources are client-run. The local crw-mcp calls them from your machine under each source's own terms, with a key of your own where the source needs one. The data never passes through fastCRW, and the call costs 0 credits.
Over REST, a client-run source answers with HTTP 409:
curl "https://api.fastcrw.com/v1/data/musicbrainz/artist?name=beatles" \
-H "Authorization: Bearer $CRW_API_KEY"
HTTP 409
{"success": false, "error": "[client_execution_required] This source runs inside the local crw MCP (...): npx -y crw-mcp install --api-key <your fastCRW API key>. It is not available in this client.", "errorCode": "client_execution_required"}The error names the fix: install the local server, which runs the source on your machine.
# Local server, runs every source including the ones that run on your machine
npx -y crw-mcp install --api-key <your fastCRW API key>The MusicBrainz source is listed on its own page: MusicBrainz API on Pergamon. Read its licence line before you use it in a product, because its terms are the source's own.
Cost model
- Catalogue, find and manifest calls are free.
- A server source call costs the item's
creditsCost. That is 1 for nearly every capability, so check the value for the capability you use. - A failed call is not billed. An upstream 4xx, 5xx or timeout returns
success: falsewith nocredits_used. - Before a batch, multiply the number of calls by the
creditsCostof each capability to estimate the spend. Read the value from the find result, not from this post, because it can change. - A client-run call costs 0 credits.
- Over MCP, a result over 256 KiB is refused with
result_too_large. If the source runs on fastCRW, that call is refunded.
Plan prices are on the pricing page.
When not to use it
Pergamon is not an index of the whole web. It calls the official API of each source in its catalogue, so a source that is not listed cannot be reached through it. It also does not write to any source. If the data you need sits on a page with no API at all, use a scraping endpoint, which fetches a page you name. That is a different job from calling a source's API.
If the source you need is missing, propose it as a provider. For account or billing questions, write to support.
Next steps
Browse the Pergamon catalogue, or get an API key and run the first call in this post.
Start with the find call above and pick one source you already parse by hand. Replace the parser with one get_data call, keep the attributionText, and compare the output. The provider pages list every capability and its options: SEC EDGAR, NVD, OpenAlex and World Bank are good first stops. The full reference is in the data sources documentation.
If you want an agent to use the catalogue from inside Claude Code, Cursor or another MCP client, the companion article on one MCP server for public data covers the two data tools.
Sources
- fastCRW data sources documentation
- SEC: Accessing EDGAR data
- NIST NVD: Start here
- NCBI E-utilities documentation
- MusicBrainz API documentation
Related: Web context layer for AI agents · Best LLM-ready web data APIs · Search API for AI agents · Pergamon catalogue