Skip to main content
Tutorial

City and Government Open Data APIs: 270+ Portals, One Key

Query Socrata, CKAN and national open data portals like NYC, San Francisco, data.gov.uk and data.gouv.fr through one API instead of a client per portal.

fastcrw
By RecepNovember 14, 20267 min read

By the fastCRW team · Catalogue figures checked 2026-11-05 against GET /v1/data · Verify independently before relying on any figure.

City and national open data portals each publish their own catalogue and their own query interface. Most people end up writing one client per portal. The Pergamon catalogue puts more than 270 city, state and national open data portals behind one fastCRW API key, so a civic-tech or journalism agent can query a city's rows, search a national catalogue and read the licence on the same response shape. This guide shows the calls that matter and the limits to plan for.

The portal zoo

Developers who try to pull data from open data portals keep hitting the same two walls. The first is a catalogue that returns metadata but no data. The data.gov catalogue page says: "data.gov and the data.gov CKAN API only contain metadata about datasets" (catalog.data.gov). The second is a default page size that quietly hides the rest. An Open Data Stack Exchange question from 2014 puts the second complaint plainly: "The result is only 10 datasets" (opendata.stackexchange.com).

Each portal has its own names for the same ideas: a dataset ID, a row filter, a sort order and a page size. The calls below use the names each portal's catalogue entry lists, so you can copy them directly.

Find a portal before you write a query

Discovery is free. Filter the catalogue to the Open data portals category, then read the capability list for the portal you want. That category holds more than 270 portals. This call looks for open data capabilities in it:

curl "https://api.fastcrw.com/v1/data/find?query=open%20data&category=Open%20data%20portals&limit=3" \
  -H "Authorization: Bearer $CRW_API_KEY"

The filter is case-insensitive, and the find response includes provider, capability, creditsCost and each option's required and optional names. Use those option names in the data call, not names you guess from another portal.

Query rows in a city portal: NYC 311 requests

NYC Open Data publishes 311 service requests as dataset erm2-nwe9. This query asks for the five most recent requests in Brooklyn, newest first. The SoQL-style where and order values are URL-encoded, and in the HTML the & separators are written as entities:

curl "https://api.fastcrw.com/v1/data/nycopendata/query?dataset=erm2-nwe9&where=borough%3D%27BROOKLYN%27&order=created_date%20DESC&limit=5" \
  -H "Authorization: Bearer $CRW_API_KEY"

In our run the response returned 5 rows. The first row's created_date was 2026-11-04T02:04:46.000 and its borough was BROOKLYN. Rows carry fields such as complaint_type, status, incident_zip and location_type. All four options on query are required: dataset, where, order and limit.

Aggregate in SoQL: San Francisco incidents by category

The San Francisco portal (datasf) takes a soql query, so the portal does the aggregation and you get a small answer back. This query counts incidents per category in dataset wg3w-h783. With the Python SDK:

# pip install crw
from crw import CrwClient

client = CrwClient()  # reads CRW_API_KEY
soql = "SELECT incident_category, count(*) GROUP BY incident_category ORDER BY count(*) DESC LIMIT 5"
result = client.get_data("datasf", "query", {"dataset": "wg3w-h783", "soql": soql})
for row in result["data"]:
    print(row["incident_category"], row["count"])

Output from our run, first two rows: Larceny Theft 309886 and Other Miscellaneous 74830. The portal returns counts as strings, so convert them before you do arithmetic. These are counts from the portal's current data, and a later run can differ.

More city portals

Seattle's building permits are a single call with a row limit, and the response is a list of permit records. In our run the call returned 3 rows. The fields include permitnum, permitclass, statuscurrent, estprojectcost and housingunits:

curl "https://api.fastcrw.com/v1/data/seattleopendata/building-permits?limit=3" \
  -H "Authorization: Bearer $CRW_API_KEY"

Three more city portals from the catalogue. For each, list its capabilities with find, then call the one that matches your question:

  • Austin (data-austintexas-gov): row-count?id=h5h5-77ye returned [{"count": "292"}] in our run.
  • Milan (dati-comune-milano-it): dataset-suggest?q=ambiente returned 10 matches in our run.
  • São Paulo (dados-prefeitura-sp-gov-br): dataset-suggest?q=educacao returned 2 matches in our run.

Milan and São Paulo publish in Italian and Portuguese. Match on dataset names and IDs in the response, not on English titles, and translate only what you show to users.

National catalogues: data.gov.uk and data.gouv.fr

National catalogues answer discovery questions, and they link out to the files. data.gov.uk's dataset search takes a q term and a row count. In our run the result count for air quality was 462 datasets:

curl "https://api.fastcrw.com/v1/data/datagovuk/datasets?q=air%20quality&rows=3" \
  -H "Authorization: Bearer $CRW_API_KEY"

data.gouv.fr is the French national platform. Its suggest capability takes q and returned 10 matches for transport in our run:

curl "https://api.fastcrw.com/v1/data/datagouvfr/datasets-suggest?q=transport" \
  -H "Authorization: Bearer $CRW_API_KEY"

Discover datasets across Socrata

Socrata's Discovery API searches the metadata of many portals that run on Socrata. Use it when you do not yet know which city publishes the dataset you need:

curl "https://api.fastcrw.com/v1/data/socrata/search?q=building%20permits&limit=3" \
  -H "Authorization: Bearer $CRW_API_KEY"

The licence on this source is per record, not per catalogue. The catalogue row says: "No blanket licence covers this catalogue." Each result carries its own licence field set by the publishing agency, and the linked portal's terms govern reuse of the data behind it. The row marks commercial use and redistribution as false for the catalogue metadata, so do not assume a commercial reuse right from a search hit.

Keep a copy in sync without re-pulling everything

Store the dataset ID, the query you ran, a hash of the response body and the time you synced it. On the next run, call the same query again and compare the hash. Re-download only the datasets whose hash changed.

Where the portal supports a date filter, narrow the query to new rows. In our run this returned the two newest NYC requests after 2026-11-01, both dated 2026-11-04. A daily job can use the same filter with its last sync time:

curl "https://api.fastcrw.com/v1/data/nycopendata/query?dataset=erm2-nwe9&where=created_date%20%3E%20%272026-11-01T00%3A00%3A00%27&order=created_date%20DESC&limit=2" \
  -H "Authorization: Bearer $CRW_API_KEY"

Keep the sync state in your own database. The catalogue does not store your copies, and it does not tell you which of your datasets changed.

What this does not do

  • There is no portal-wide licence. Read the licence on each response, and on each dataset's own page, before you republish data.
  • Data quality is the publisher's. A portal can publish late, partial or inconsistent rows. Check counts against a second source before you report them.
  • A row limit is not a full export. Page through large datasets with the portal's own paging options rather than assuming one call returns everything.
  • Portal names are not stable IDs. Some portals are added or switched off over time. Check the provider page before you hard-code a portal into a product.

Start from the question, then pick the portal

Most civic questions have a local answer and a national one. A housing agent asking about building permits in one city needs that city's portal. A policy agent asking about air quality across a country needs the national catalogue first, then the local portal that holds the detail. Start with the question, use find with the Open data portals category to see which portals hold the data, and only then pick the calls.

Portals can also disagree. The same kind of record can be counted differently in two cities because each publisher defines its fields on its own. Compare definitions in the dataset description before you compare numbers across portals.

Response shapes differ by portal

The same kind of question comes back in different shapes. Some portals return a bare array of rows, some wrap the answer in a CKAN-style object, and some add timing and warning fields. These are the shapes we saw on 2026-11-05:

CapabilityTop-level shape in our run
nycopendata/queryArray of rows
datasf/queryArray of rows, counts returned as strings
seattleopendata/building-permitsArray of rows
datagovuk/datasetsObject with help, result and success
socrata/searchObject with resultSetSize, results, timings and warnings
datagouvfr/datasets-suggestArray
data-austintexas-gov/row-countArray with one object, count as a string
dati-comune-milano-it/dataset-suggestObject with help, result and success

Write one small adapter per shape, not one per portal. Two portals that share a shape can share the parsing code, and only the query options differ between them.

Check a dataset before you use it

A portal response is only as current and complete as the publisher made it. Before a number leaves your pipeline, run four checks:

  • Licence. Read provenance.licence on that response, and the dataset's own page. Do not reuse a licence you saw on a different dataset in the same portal.
  • Freshness. Read provenance.retrieved_at and is_stale. A stale flag means the body came from fastCRW's cache, and cache_age_seconds says how old that copy is.
  • Completeness. Where a portal has a row-count capability, compare it with the rows you pulled. The Austin row-count call returned 292 for dataset h5h5-77ye in our run.
  • Reproducibility. Store the full query string with your output. Someone who reads your number should be able to run the same call.

Errors you will see

  • Unknown name. A wrong provider or capability returns HTTP 404 with errorCode unknown_data_source. Run find with the Open data portals category to get the exact names.
  • Upstream failure. A portal that fails returns success: false with upstream_error. A portal that times out returns upstream_timeout. A failed call is not billed. Retry later, and do not loop on a portal that is down.
  • Empty answers are not errors. A search with no matches returns an empty result, as the Milan dataset-suggest call does for a query without hits. Treat an empty array as "no match", not as a failure.

Sources

Related: Congress.gov and Federal Register API for policy agents · Regulatory compliance monitoring · Public data API for AI agents · Pergamon catalogue

FAQ

Frequently asked questions

Should I scrape a city open data portal or use its API?
Use the portal's query interface first. Rows, SoQL aggregates and dataset searches return structured data with the licence attached, which is cheaper and more reliable than parsing a rendered page.
How do I find the right dataset in a portal I have not used before?
Run find with category Open data portals and a short query, then call the portal's suggest or search capability with your own term. Match on dataset names and IDs when the portal publishes in another language.
Can I republish open government data in a product?
Check the licence on each response and on each dataset's page. Catalogues mix public domain, Creative Commons and bare disclaimers, so there is no single answer for a whole portal. The Socrata discovery row explicitly carries no blanket licence.
How do I keep a local copy of a city dataset up to date?
Store the dataset ID, the query, a hash of the response and the sync time. Re-run the query on a schedule, compare hashes, and where the portal supports a date filter, pull only the rows newer than your last sync.

Get Started

Try fastCRW free

Run a live request in the playground — no signup required. Or grab a free API key with 500 credits, earn up to 500 more, no credit card.

Continue exploring

More tutorial posts

View category archive