How do I stream scrape results with Server-Sent Events in Node.js?

Open an HTTP handler that sets Content-Type: text/event-stream, flushes headers, and never ends the response until the job finishes. Start an async crawl with POST /v1/crawl to get a job ID, then poll GET /v1/crawl/:id server-side on an interval, diff against what you have already sent, and write each new page as an SSE frame (an event: line, a data: line with JSON, then a blank line). The browser consumes it with the native EventSource API. Add a comment-line heartbeat every ~15s to keep proxies from closing the idle socket.

SSE vs WebSockets for crawl progress: which should I use?

Use SSE. Crawl progress is one-directional — the server has new pages, the client only renders them — which is exactly what Server-Sent Events were designed for. SSE rides on plain HTTP, auto-reconnects via EventSource, and traverses existing proxies and load balancers with no extra protocol or port. WebSockets are full duplex and worth it only when the client also streams data back, which crawl progress does not (cancellation is a one-shot DELETE, not a stream).

How do I show live progress for a long-running crawl?

Emit a progress event carrying completed/total on every server-side poll, alongside the per-page events. A moving denominator turns a long gap between pages into '37 of 100 done, still working' rather than a frozen spinner. In fast mode, fastCRW's p90 is 4348 ms — the lowest of three tools tested (diagnose_3way.py, 819 labeled URLs, 2026-05-08) — and the chrome-stealth fallback that recovers pages others miss also delivers this strong tail. SSE lets the fast pages render immediately while recoveries fill in as visible progress.

How do I cancel a running crawl job?

Call DELETE /v1/crawl/:id with the job ID returned by POST /v1/crawl. Wire it into both an explicit cancel action and your SSE handler's req.on('close') callback, so when the user closes the tab you stop the server-side polling loop and cancel the underlying crawl — otherwise you leak an open job and keep burning credits for an abandoned page.

Does the fastCRW engine push results over SSE natively?

No. The engine is stateless per request and exposes a job-polling API — POST /v1/crawl returns a job ID, GET /v1/crawl/:id returns status and results, DELETE /v1/crawl/:id cancels. You build the SSE session in your own Node.js tier, bridging that polling surface to the browser. The engine does offer a POST /mcp Streamable HTTP transport for MCP clients, but that is a separate concern from a browser-facing SSE feed.

Streaming Scrape Results in Node.js with SSE

By the fastCRW team · Engine endpoints and benchmark figures verified 2026-05-18 against the canonical fact sheet · Verify independently before relying on any number.

Why stream scrape results with SSE in Node.js

Node.js web scraping with SSE streaming solves a specific UX problem: a whole-site crawl can take minutes, and a UI that shows a frozen spinner for that long looks broken even when the job is healthy. Server-Sent Events (SSE) let your Node.js server push each completed page to the browser the moment it lands, so the user watches results accumulate in real time instead of staring at nothing. This post builds a concrete pattern: a Node.js SSE endpoint that drives a Firecrawl-compatible async crawl, polls the job server-side, and emits deltas to the client — with backpressure, heartbeats, and clean cancellation handled.

The honest framing up front: the fastCRW engine itself does not push results over SSE. It exposes a job-polling API — POST /v1/crawl returns a job ID, GET /v1/crawl/:id returns status plus results, and DELETE /v1/crawl/:id cancels. SSE is something you build in your Node.js tier on top of that polling surface. That is the right boundary anyway: the engine stays stateless per request, and your app owns the streaming session.

Polling vs Server-Sent Events vs WebSockets

There are three common ways to get crawl progress to a browser:

Client polling. The browser hits your API every few seconds asking "done yet?". Simple, but wasteful, laggy, and it hammers your server with redundant requests.
WebSockets. Full duplex, great when the client also streams data back. For one-directional progress updates it is overkill: you take on a separate protocol, sticky-session concerns, and a heavier client.
Server-Sent Events. A single long-lived HTTP response of text/event-stream. The server pushes; the browser consumes via the native EventSource API, which auto-reconnects for free. For crawl progress — strictly server-to-client — SSE is the lean fit.

When SSE is the right fit for crawl progress

Crawl progress is one-directional: the server has new pages, the client only needs to render them. There is no client-to-server stream once the job is kicked off (cancellation is a one-shot DELETE, not a stream). That is exactly the shape SSE was designed for. You get reconnection, event IDs, and plain HTTP — no new protocol, no extra port, and it traverses proxies and load balancers that already speak HTTP/1.1.

The UX win: results as they arrive

The payoff is psychological as much as technical. A crawl that surfaces page 1 at 400 ms, page 2 at 1.2 s, and page 40 a minute later feels fast and alive, even if total wall-clock is identical to a batch response. Streaming turns dead time into visible progress.

Building an SSE endpoint in Node.js

An SSE endpoint is just an HTTP handler that never calls res.end() until it is done — it keeps the socket open and writes framed messages.

text/event-stream headers and the response loop

The three things that make a response an SSE stream are the content type, disabling buffering, and keeping the connection alive. In your Express handler, call res.writeHead(200, ...) with "Content-Type": "text/event-stream", "Cache-Control": "no-cache, no-transform", Connection: "keep-alive", and "X-Accel-Buffering": "no", then res.flushHeaders(). A small send(event, data) helper writes one frame per call.

Each message is a line of event: <name> (optional), then data: <json>, then a blank line to terminate the frame — concretely res.write('event: page\n'); res.write('data: ' + JSON.stringify(payload) + '\n\n');. The X-Accel-Buffering: no header matters in production: without it, a reverse proxy like nginx will buffer your stream and the client sees nothing until the buffer flushes, which defeats the entire point.

Heartbeats and reconnection with Last-Event-ID

Idle connections get reaped by proxies and load balancers, usually after 30–60 seconds of silence. During a slow page that can happen mid-crawl. Send a comment-line heartbeat on an interval to keep the socket warm — const heartbeat = setInterval(() => res.write(':hb\n\n'), 15000);.

A line starting with : is an SSE comment — ignored by the client but enough to keep intermediaries from closing the connection. For reconnection, attach an id: to each event; on a dropped connection the browser's EventSource automatically resends the last seen ID in the Last-Event-ID request header, so your handler can resume from where it left off rather than replaying the whole crawl.

Handling backpressure and client disconnects

Two failure modes will bite you if you ignore them. First, backpressure: res.write() returns false when the kernel send buffer is full (a slow client on mobile, say). Respect it — pause your producer until the drain event rather than buffering unbounded pages in memory. Second, client disconnects: when the user closes the tab, the request fires a close event. Listen for it and tear everything down, or you will leak a polling loop and an open crawl job for every abandoned page. In req.on("close", ...) you should clearInterval(heartbeat), stop the GET /v1/crawl/:id polling loop, and fire DELETE /v1/crawl/:id to cancel the job.

Streaming a crawl job's progress

With the SSE plumbing in place, the data source is a Firecrawl-compatible async crawl. The lifecycle is: start, poll, push deltas, finish or cancel.

Starting an async crawl and getting a job ID

A crawl is asynchronous by design — you do not block an HTTP request for two minutes. POST /v1/crawl with a body like { url, maxDepth: 2, maxPages: 100 } and a Bearer API key returns immediately with a job ID, which you read off the JSON response as id.

Because the API is Firecrawl-compatible, this is a base-URL swap away from an existing Firecrawl integration — point at your own self-hosted engine or the managed cloud without rewriting the client. Note the crawl caps: maxDepth tops out at 10 and maxPages at 1000, so size the job to what your stream can comfortably render.

Polling /v1/crawl/:id server-side, pushing deltas over SSE

Here is the bridge: the server polls the job, diffs against what it has already sent, and pushes only the new pages down the SSE stream. The browser never polls — your server does, once, on the client's behalf. A pump() function runs on a setTimeout loop (a ~1.5-second cadence works well): it does GET /v1/crawl/:id, which returns { status, completed, total, data: [...] }; for each page in job.data.slice(sentCount) it calls send("page", { url, markdown }); it advances sentCount = job.data.length and emits send("progress", { completed, total }); and when status is "completed" or "failed" it sends a done event, calls res.end(), and stops the loop.

The slice(sentCount) is the whole trick: you emit each completed page exactly once. A 1.5-second server-side poll is invisible to the user because every poll that finds new pages immediately fans them out over the already-open SSE socket.

Cancelling a crawl with DELETE /v1/crawl/:id

If the user navigates away or hits a cancel button, do not just drop the SSE connection — also cancel the underlying job so you stop burning credits and engine time. The DELETE /v1/crawl/:id endpoint stops the BFS crawl. Wire it into both the explicit cancel action and the req.on("close") handler shown earlier so an abandoned tab cleans up automatically.

A working Node.js + SSE scrape pipeline

Putting the pieces together, the full handler is small: open the stream, start the crawl, pump deltas, and tear down on completion or disconnect.

Express route that drives a Firecrawl-compatible crawl

The route does four things in order — set SSE headers, POST /v1/crawl to get a job ID, run the pump() loop that polls and pushes, and register cleanup on close. Everything above slots into one ~50-line handler. Keep the API key server-side; the browser only ever talks to your SSE endpoint, never to the crawl API directly.

Emitting each completed page to the client

On the browser side, consumption is the native EventSource — no library needed. Construct it with new EventSource("/crawl/stream?url=" + encodeURIComponent(url)), then register listeners: es.addEventListener("page", e => appendRow(JSON.parse(e.data))), es.addEventListener("progress", e => updateBar(JSON.parse(e.data))), and es.addEventListener("done", () => es.close()).

Each page event appends a row; each progress event nudges a progress bar. The user sees the crawl fill in live.

Closing the stream cleanly on job completion

When the job reaches completed or failed, send a final done event and call res.end() server-side; the client closes its EventSource in the done handler. Without the explicit es.close(), EventSource would treat the server hang-up as a dropped connection and try to reconnect, restarting a finished crawl. Always signal completion in-band.

Latency, the long tail, and timeouts

Streaming is not just polish — it is a direct response to the engine's real latency profile, and being honest about that profile is what makes the pattern worth building.

Why the latency distribution shapes your stream UX

On the canonical 3-way scrape benchmark (diagnose_3way.py, Firecrawl's public dataset, 819 labeled URLs, 2026-05-08), fastCRW's p50 latency is 1914 ms, which beats Firecrawl's 2305 ms. In fast mode, fastCRW's p90 is 4348 ms — the lowest of the three tools tested (Crawl4AI 4754 ms, Firecrawl 6937 ms). The chrome-stealth fallback that recovers pages the other tools miss — the same mechanism behind fastCRW's highest truth-recall of the three, 63.74% of 819 labeled URLs — is what delivers both the recall lead and the strong fast-mode tail. The pages the engine fights hardest to recover are frequently the most valuable ones in your crawl.

That distribution is precisely why streaming matters. With SSE, the fast pages render in the first second or two and the recoveries fill in as they finish — even slow-rendering pages become visible progress instead of a frozen UI.

Showing progress while slow pages finish

Lean on the progress event. Emitting completed / total on every poll gives the user a moving denominator, so a long gap between pages reads as "37 of 100 done, still working" rather than "stuck". Pair that with a per-page timestamp in the row and the tail is fully legible. See scraping latency explained for the full p50/p90/p99 breakdown and why a single average would be misleading here.

Stateless requests: designing for reconnection

Two limits to design around. First, the engine is stateless per request — there is no server-pushed event channel from the engine itself, which is why the SSE session lives in your Node.js tier and bridges the job-polling API to the browser. Second, if your Node.js process restarts mid-crawl, the SSE connection drops; the job keeps running engine-side, so a reconnect that re-issues GET /v1/crawl/:id with the saved job ID can resume cleanly. Persist the job ID, not the in-memory stream. For agent and MCP clients, the engine does expose a POST /mcp Streamable HTTP transport, but that is an MCP concern, not a browser SSE feed — do not conflate the two.

Sources

fastCRW canonical fact sheet — endpoint table (/v1/crawl, GET/DELETE /v1/crawl/:id, /mcp) and scrape benchmark (diagnose_3way.py, 819 labeled URLs, 2026-05-08): github.com/us/crw
MDN — Server-Sent Events and the EventSource API: developer.mozilla.org
fastCRW managed cloud: fastcrw.com