By the fastCRW team · Catalogue figures checked 2026-11-05 against GET /v1/data · Verify independently before relying on any figure.
City and national open data portals each publish their own catalogue and their own query interface. Most people end up writing one client per portal. The Pergamon catalogue puts more than 270 city, state and national open data portals behind one fastCRW API key, so a civic-tech or journalism agent can query a city's rows, search a national catalogue and read the licence on the same response shape. This guide shows the calls that matter and the limits to plan for.
The portal zoo
Developers who try to pull data from open data portals keep hitting the same two walls. The first is a catalogue that returns metadata but no data. The data.gov catalogue page says: "data.gov and the data.gov CKAN API only contain metadata about datasets" (catalog.data.gov). The second is a default page size that quietly hides the rest. An Open Data Stack Exchange question from 2014 puts the second complaint plainly: "The result is only 10 datasets" (opendata.stackexchange.com).
Each portal has its own names for the same ideas: a dataset ID, a row filter, a sort order and a page size. The calls below use the names each portal's catalogue entry lists, so you can copy them directly.
Find a portal before you write a query
Discovery is free. Filter the catalogue to the Open data portals category, then read the capability list for the portal you want. That category holds more than 270 portals. This call looks for open data capabilities in it:
curl "https://api.fastcrw.com/v1/data/find?query=open%20data&category=Open%20data%20portals&limit=3" \
-H "Authorization: Bearer $CRW_API_KEY"
The filter is case-insensitive, and the find response includes provider, capability, creditsCost and each option's required and optional names. Use those option names in the data call, not names you guess from another portal.
Query rows in a city portal: NYC 311 requests
NYC Open Data publishes 311 service requests as dataset erm2-nwe9. This query asks for the five most recent requests in Brooklyn, newest first. The SoQL-style where and order values are URL-encoded, and in the HTML the & separators are written as entities:
curl "https://api.fastcrw.com/v1/data/nycopendata/query?dataset=erm2-nwe9&where=borough%3D%27BROOKLYN%27&order=created_date%20DESC&limit=5" \
-H "Authorization: Bearer $CRW_API_KEY"
In our run the response returned 5 rows. The first row's created_date was 2026-11-04T02:04:46.000 and its borough was BROOKLYN. Rows carry fields such as complaint_type, status, incident_zip and location_type. All four options on query are required: dataset, where, order and limit.
Aggregate in SoQL: San Francisco incidents by category
The San Francisco portal (datasf) takes a soql query, so the portal does the aggregation and you get a small answer back. This query counts incidents per category in dataset wg3w-h783. With the Python SDK:
# pip install crw
from crw import CrwClient
client = CrwClient() # reads CRW_API_KEY
soql = "SELECT incident_category, count(*) GROUP BY incident_category ORDER BY count(*) DESC LIMIT 5"
result = client.get_data("datasf", "query", {"dataset": "wg3w-h783", "soql": soql})
for row in result["data"]:
print(row["incident_category"], row["count"])
Output from our run, first two rows: Larceny Theft 309886 and Other Miscellaneous 74830. The portal returns counts as strings, so convert them before you do arithmetic. These are counts from the portal's current data, and a later run can differ.
More city portals
Seattle's building permits are a single call with a row limit, and the response is a list of permit records. In our run the call returned 3 rows. The fields include permitnum, permitclass, statuscurrent, estprojectcost and housingunits:
curl "https://api.fastcrw.com/v1/data/seattleopendata/building-permits?limit=3" \
-H "Authorization: Bearer $CRW_API_KEY"
Three more city portals from the catalogue. For each, list its capabilities with find, then call the one that matches your question:
- Austin (
data-austintexas-gov):row-count?id=h5h5-77yereturned[{"count": "292"}]in our run. - Milan (
dati-comune-milano-it):dataset-suggest?q=ambientereturned 10 matches in our run. - São Paulo (
dados-prefeitura-sp-gov-br):dataset-suggest?q=educacaoreturned 2 matches in our run.
Milan and São Paulo publish in Italian and Portuguese. Match on dataset names and IDs in the response, not on English titles, and translate only what you show to users.
National catalogues: data.gov.uk and data.gouv.fr
National catalogues answer discovery questions, and they link out to the files. data.gov.uk's dataset search takes a q term and a row count. In our run the result count for air quality was 462 datasets:
curl "https://api.fastcrw.com/v1/data/datagovuk/datasets?q=air%20quality&rows=3" \
-H "Authorization: Bearer $CRW_API_KEY"
data.gouv.fr is the French national platform. Its suggest capability takes q and returned 10 matches for transport in our run:
curl "https://api.fastcrw.com/v1/data/datagouvfr/datasets-suggest?q=transport" \
-H "Authorization: Bearer $CRW_API_KEY"
Discover datasets across Socrata
Socrata's Discovery API searches the metadata of many portals that run on Socrata. Use it when you do not yet know which city publishes the dataset you need:
curl "https://api.fastcrw.com/v1/data/socrata/search?q=building%20permits&limit=3" \
-H "Authorization: Bearer $CRW_API_KEY"
The licence on this source is per record, not per catalogue. The catalogue row says: "No blanket licence covers this catalogue." Each result carries its own licence field set by the publishing agency, and the linked portal's terms govern reuse of the data behind it. The row marks commercial use and redistribution as false for the catalogue metadata, so do not assume a commercial reuse right from a search hit.
Keep a copy in sync without re-pulling everything
Store the dataset ID, the query you ran, a hash of the response body and the time you synced it. On the next run, call the same query again and compare the hash. Re-download only the datasets whose hash changed.
Where the portal supports a date filter, narrow the query to new rows. In our run this returned the two newest NYC requests after 2026-11-01, both dated 2026-11-04. A daily job can use the same filter with its last sync time:
curl "https://api.fastcrw.com/v1/data/nycopendata/query?dataset=erm2-nwe9&where=created_date%20%3E%20%272026-11-01T00%3A00%3A00%27&order=created_date%20DESC&limit=2" \
-H "Authorization: Bearer $CRW_API_KEY"
Keep the sync state in your own database. The catalogue does not store your copies, and it does not tell you which of your datasets changed.
What this does not do
- There is no portal-wide licence. Read the licence on each response, and on each dataset's own page, before you republish data.
- Data quality is the publisher's. A portal can publish late, partial or inconsistent rows. Check counts against a second source before you report them.
- A row limit is not a full export. Page through large datasets with the portal's own paging options rather than assuming one call returns everything.
- Portal names are not stable IDs. Some portals are added or switched off over time. Check the provider page before you hard-code a portal into a product.
Start from the question, then pick the portal
Most civic questions have a local answer and a national one. A housing agent asking about building permits in one city needs that city's portal. A policy agent asking about air quality across a country needs the national catalogue first, then the local portal that holds the detail. Start with the question, use find with the Open data portals category to see which portals hold the data, and only then pick the calls.
Portals can also disagree. The same kind of record can be counted differently in two cities because each publisher defines its fields on its own. Compare definitions in the dataset description before you compare numbers across portals.
Response shapes differ by portal
The same kind of question comes back in different shapes. Some portals return a bare array of rows, some wrap the answer in a CKAN-style object, and some add timing and warning fields. These are the shapes we saw on 2026-11-05:
| Capability | Top-level shape in our run |
|---|---|
nycopendata/query | Array of rows |
datasf/query | Array of rows, counts returned as strings |
seattleopendata/building-permits | Array of rows |
datagovuk/datasets | Object with help, result and success |
socrata/search | Object with resultSetSize, results, timings and warnings |
datagouvfr/datasets-suggest | Array |
data-austintexas-gov/row-count | Array with one object, count as a string |
dati-comune-milano-it/dataset-suggest | Object with help, result and success |
Write one small adapter per shape, not one per portal. Two portals that share a shape can share the parsing code, and only the query options differ between them.
Check a dataset before you use it
A portal response is only as current and complete as the publisher made it. Before a number leaves your pipeline, run four checks:
- Licence. Read
provenance.licenceon that response, and the dataset's own page. Do not reuse a licence you saw on a different dataset in the same portal. - Freshness. Read
provenance.retrieved_atandis_stale. A stale flag means the body came from fastCRW's cache, andcache_age_secondssays how old that copy is. - Completeness. Where a portal has a row-count capability, compare it with the rows you pulled. The Austin
row-countcall returned 292 for dataseth5h5-77yein our run. - Reproducibility. Store the full query string with your output. Someone who reads your number should be able to run the same call.
Errors you will see
- Unknown name. A wrong provider or capability returns HTTP 404 with
errorCodeunknown_data_source. Runfindwith the Open data portals category to get the exact names. - Upstream failure. A portal that fails returns
success: falsewithupstream_error. A portal that times out returnsupstream_timeout. A failed call is not billed. Retry later, and do not loop on a portal that is down. - Empty answers are not errors. A search with no matches returns an empty result, as the Milan
dataset-suggestcall does for a query without hits. Treat an empty array as "no match", not as a failure.
Sources
- CKAN API documentation
- Socrata developer documentation
- NYC Open Data
- data.gov.uk
- data.gouv.fr API documentation
Related: Congress.gov and Federal Register API for policy agents · Regulatory compliance monitoring · Public data API for AI agents · Pergamon catalogue
