Real Estate Market Data Pipeline: A Build Guide

By the fastCRW team · Benchmark and credit figures verified against the canonical fact sheet 2026-05-29 · Verify independently before quoting internally.

What a real estate market scraping data pipeline produces

A real estate market scraping data pipeline turns a set of listing portals into a refreshable table of structured property records. The output is not HTML or markdown — it is rows: one per listing, each with the fields your comps model or dashboard actually consumes. The pipeline has three moving parts: a crawler that discovers and fetches listing pages, an extractor that pulls a fixed schema out of each page, and a scheduler that re-runs the whole thing so the dataset reflects today's market instead of last month's.

This guide builds that pipeline on fastCRW — a single Rust binary with a Firecrawl-compatible REST API (/v1/map, /v1/crawl, /v1/scrape, /v1/search) — but the pattern is portable. The decisive concerns for market data are coverage (did we miss listings?), cost (recurring portal-scale crawls add up), and freshness (is the snapshot current?). We will address each, and state plainly where this approach has limits.

Listing fields: price, beds, location, status

Pick a flat schema before you write a single request. A workable minimum for most market-analytics use cases:

• price — the numeric asking price, currency-normalized.
• beds / baths — integers; many portals render these as icons, so plan for occasional nulls.
• location — address or at least neighborhood + postal code, for geocoding downstream.
• status — active / pending / sold; this is what makes the dataset a market feed rather than a static directory.
• sourceUrl and scrapedAt — provenance you store yourself, not from the page.

Keep it flat. Nested objects per listing make diffing and database loads harder for no analytical gain at this stage.

One-time dataset vs continuous market feed

Be honest about which you are building. A one-time dataset is a single crawl pass you load and analyze once — fine for a market study or a model's training snapshot. A continuous feed re-crawls on a schedule and tracks what changed: new listings, price cuts, status flips to "sold." The continuous feed is where the real value lives for proptech, and it is where cost discipline and recall matter most, because errors compound across every refresh.

List crawling property sites

Map and crawl listing index pages

Start by discovering URLs, not by guessing them. POST /v1/map returns the link graph of a site so you can see how listing detail pages are addressed before committing crawl budget. Then POST /v1/crawl runs an async breadth-first crawl from a seed (typically a search-results or category index page) and returns a job ID you poll with GET /v1/crawl/:id. Bound the job explicitly: maxDepth (capped at 10) keeps the crawl from wandering into unrelated sections, and maxPages (capped at 1000 per job) is your hard ceiling. For a portal with more than 1,000 relevant pages, segment by region or price band and run multiple bounded jobs rather than one unbounded one. This is the same crawl mechanics covered in crawling an entire website from its sitemap.

Handling pagination and infinite scroll

Listing portals paginate in two ways. Classic numbered pagination exposes ?page=2-style URLs that a crawl discovers and follows naturally — this is the easy case. Infinite scroll is harder: the next batch of listings loads via XHR as you scroll, so there is no link in the initial HTML for the crawler to follow. Two honest tactics: (1) find the underlying paginated API the scroll calls and crawl that directly — often the cleanest path; or (2) use the chrome renderer to execute the page's JavaScript, accepting that it is slower. There is no magic "scroll forever" button; you are choosing between hitting the data API and paying for a real browser. See the limitations writeup for where the JS path genuinely struggles.

Extracting repeated records with a JSON schema

Once you have a listing page's content, extraction is a schema-guided LLM pass. Send formats: ["json"] with a jsonSchema on the scrape request and the engine returns structured fields instead of markdown. For repeated records — an index page that lists many properties — define the schema as an array of listing objects so one request yields many rows. The full pattern, including schema design and validation, is in structured extraction with a JSON schema and the broader list-crawling for structured data guide. Two cost facts to internalize now: any request with formats: ["json"] costs 5 credits (it invokes the extraction model), while a plain crawl page is 1 credit regardless of renderer. That gap drives the architecture below. LLM-based JSON extraction is a managed feature available on paid plans.

Keeping the dataset fresh

Scheduled crawls for new and changed listings

A market feed is a crawl on a timer. Run the same bounded crawl jobs nightly (or hourly for hot markets) from cron, a CI scheduler, or a workflow tool — the mechanics are in scheduled crawls with cron. The cadence is a cost-versus-freshness trade you tune per portal: a luxury market that turns over slowly does not need hourly passes; a high-velocity rental board might. Crawl the index pages every run to catch new listings; you do not need to re-extract every detail page every time if the index already surfaces price and status.

Diffing against the prior snapshot

Freshness is meaningless without a diff. Key each listing by a stable identifier — the portal's listing ID, or a hash of sourceUrl if no ID is exposed — and compare each run against the last stored snapshot. Three change classes matter: new (ID not seen before), changed (price or status differs), and gone (previously present, now absent — often a sale or delisting). The "gone" class is the one teams forget, and it is exactly the signal a comps model needs.

Storing history yourself (stateless engine)

fastCRW is stateless per request — it does not remember yesterday's crawl, hold a session, or store snapshots for you. That is a deliberate boundary, not an oversight: the engine fetches and extracts; persistence and history are your pipeline's job. In practice you write each run's rows to your own store (Postgres, a warehouse, even partitioned Parquet) with a scrapedAt timestamp, and the diff reads the previous partition. The upside of statelessness is that you own the history format and retention completely; the cost is that you must build the storage layer rather than query a vendor's.

Accuracy and missing listings

Why recall matters for market coverage

For a market dataset, a silently dropped listing is worse than a slow one — it is a hole in your comps that no error log flags. This is why we lead with recall. On Firecrawl's public labeled dataset (diagnose_3way.py, 819 labeled URLs, 2026-05-08), fastCRW returned correct content for 63.74% of labeled URLs — the highest truth-recall of the three tools tested (Crawl4AI 59.95%, Firecrawl 56.04%). Higher recall means fewer listings quietly missing from each pass, which directly tightens market coverage. Recall is never 100% for any tool, so treat the number as "fewer holes," not "no holes," and reconcile against a known-good sample of listings periodically.

Scrape-success and zero-error framing

On the same run, fastCRW reached 91.8% scrape-success of reachable URLs with 0 thrown errors across 3,000 requests. The pairing matters: "0 errors" alone understates the result — it means no exceptions crashed the job, but combining it with the success rate shows that most pages returned usable content and the job completed predictably. On latency, the benchmark put fastCRW's p50 at 1914 ms (beating Firecrawl's 2305 ms); in fast mode its p90 is 4348 ms — the lowest of the three tools tested (Crawl4AI 4754 ms, Firecrawl 6937 ms). For a nightly batch crawl, that tail is rarely a constraint. See the benchmarks page for the full p50/p90 split.

Cost at portal scale

Credit math for large recurring crawls

Recurring crawls are where naive pricing surprises you, so do the arithmetic up front. A crawled page costs 1 credit — flat, regardless of renderer. A request that runs JSON extraction costs 5 credits. So a nightly pass over a 5,000-page portal that extracts structured fields on every page is 25,000 credits a night — about 750,000 a month — whereas crawling those 5,000 pages and extracting only the ~500 that changed (caught by your diff) is roughly 5,000 + 2,500 = 7,500 credits a night, an order of magnitude less. The single biggest cost lever in a real estate pipeline is extracting only what changed. Map first, crawl bounded, diff, then extract the delta. Current per-credit pricing is on the pricing page — derive your monthly estimate from your own page counts and refresh cadence rather than a headline tier number.

Self-host to remove per-page cloud cost

If portal-scale recurring crawls push your credit spend uncomfortably high, the AGPL-3.0 engine self-hosts as a single ~8 MB image (one container). Self-hosted, scrapes and crawls cost $0 per 1,000 — you pay only for your own server, which for a steady nightly batch can run on a small VPS. The trade is that you operate the engine, and LLM-based JSON extraction stays a managed cloud feature on paid plans rather than something the self-hosted binary runs on its own. For high-volume, predictable market feeds, owning the server often beats metered cloud; for bursty or low-volume work, managed cloud is simpler. The fact that the same engine and the same Firecrawl-compatible API run in both modes means the decision is reversible — start on cloud, move the heavy recurring jobs to self-host later without rewriting the pipeline.

Limits to plan for

JS-heavy map widgets and anti-bot

State the gaps plainly. Portals built around heavy interactive map widgets often load listings through JavaScript that the http renderer won't see — you'll need the chrome renderer (still 1 credit/page) or, better, the site's underlying data API. And fastCRW does not ship a Fire-engine-class anti-bot system: aggressively protected portals with sophisticated bot defenses may rate-limit or block you, and no scraper makes that go away. There is also no screenshot output — a request for formats: ["screenshot"] returns HTTP 422 — so if your workflow depended on listing-photo screenshots, that is not the tool for it. Where heavy managed anti-bot or agentic crawling is the hard requirement, a cloud-only vendor with those features genuinely wins; concede that rather than pretend otherwise.

Single-URL extraction means crawl or iterate

The managed /v1/extract convenience endpoint is single-URL — there is no multi-URL batched extract endpoint. For many listings you therefore do one of two things: run /v1/crawl (which fetches many pages in one job) and extract per page, or iterate /v1/scrape concurrently across your URL list. This is a pipeline-shape constraint, not a blocker — it just means "batch" is something your orchestration layer expresses as concurrency, not a single API call. Plan your scheduler around fan-out rather than a one-shot batch request.

Sources

• Scrape benchmark (truth-recall 63.74% of 819 labeled URLs, 91.8% scrape-success of reachable URLs, 0 errors, p50 1914 ms / fast-mode p90 4348 ms): fastCRW canonical fact sheet, diagnose_3way.py on Firecrawl's public dataset, 2026-05-08.
• Credit costs (crawl 1/page flat for all renderers, JSON extraction 5): fastCRW fact sheet.
• API surface (/v1/map, /v1/crawl with maxDepth ≤10 / maxPages ≤1000, single-URL /v1/extract, no screenshot output): fastCRW fact sheet,.
• Self-host footprint and $0/1,000 self-hosted cost: fastCRW fact sheet,, · github.com/us/crw

Related: List crawling for structured data · Structured extraction with a JSON schema · Crawl an entire website from its sitemap · Scheduled crawls with cron