Can I scrape behind a login without server-side sessions?

Often, yes. If the gate is a credential the server checks on each request — a session cookie, an Authorization bearer token, or an API key — you can obtain it once and forward it in the headers of each stateless scrape call. That covers most token- and cookie-authenticated dashboards and member pages. It does not cover logins that require live in-browser steps (JavaScript-derived one-time tokens, CAPTCHAs, multi-page wizards), which need a stateful session tool.

How do I send cookies with a stateless scrape request?

Attach them as request headers on each /v1/scrape call. Because the model is stateless, the cookies (and any bearer token or API key) live in your client and travel with every request rather than being stored server-side. The target sees an authenticated request and returns the gated content. This is idempotent and retry-safe: resend the identical request to any node and you get the same result.

When do I actually need stateful sessions?

When your workflow is a genuine multi-step in-browser interaction whose later steps depend on the live state of earlier ones — logging in through a JavaScript-driven form, clicking through several gated pages, dismissing modals, then extracting data that only appears after that interaction. You also need them for managed anti-bot evasion you do not want to operate yourself. In those cases a stateful session endpoint genuinely wins; for reachable-by-forwarded-credentials targets and high-throughput crawling, stateless is simpler and cheaper.

Stateless vs Stateful Scraping: Session Tradeoffs

By the fastCRW team · Architecture and capability claims verified 2026-05-18 · Verify independently before relying on them.

Disclosure: We build fastCRW, so weigh this accordingly. fastCRW is stateless per request by design, and this post lays out why that model covers the overwhelming majority of real scraping workloads.

Stateless vs stateful scraping: the core trade-off

The choice between stateless and stateful scraping comes down to one question: does your scraper hold a live browser session between API calls, or does each request stand on its own? Stateful scraping keeps a persistent session — a logged-in browser, a cookie jar, an in-progress multi-step flow — alive on the server so a sequence of calls can build on one another. Stateless scraping treats every request as self-contained: you pass in everything the request needs (the URL, headers, cookies), the engine does the work, and it forgets you the moment it responds.

Neither is universally correct. Stateful sessions unlock genuinely hard workflows that statelessness cannot reach. Stateless requests are simpler to scale, retry, and reason about. This post lays out what each model actually buys you, what it costs, and how to handle auth-gated content either way.

What stateful scraping actually means

Persistent browser sessions and cookie jars

A stateful scraper keeps a real browser context running on the server between your API calls. That context accumulates state: cookies set by the site, local-storage values, an authenticated session token, even the current scroll position or an open modal. When you make the next call, the server routes it back to that same warm context, so the page "remembers" everything that happened before. Firecrawl's /interact-style endpoint works this way — it keeps a browser session alive across calls so you can click, type, wait, and then read, all against one continuous page.

Multi-step flows and logins

The payoff of that persistence is multi-step interaction. You can submit a login form, let the site set its session cookie in the live context, navigate to a gated page, dismiss a cookie banner, paginate through an infinite-scroll feed, or complete a wizard that only reveals data on the final step. Each action mutates the server-held state, and the next action sees the result. For sites that gate content behind genuine in-browser interaction — not just a header you can replay — this is the only model that works cleanly.

The hidden cost of server-side state

Session affinity and sticky routing

The moment your server holds a live session, every follow-up call for that session must land on the exact node that holds it. That is session affinity, and it forces sticky routing: your load balancer can no longer treat nodes as interchangeable. You need a session store, a routing key, and a fallback plan for when the node holding a session dies mid-flow. What was a stateless "any node serves any request" architecture becomes a distributed-state problem.

Memory held per live session

Every open browser session pins resources. A headless Chrome context is not free — the opt-in chrome configuration in fastCRW's own stack is roughly a 500 MB image with about 1 GB resident when running. Multiply that by every concurrent live session a stateful design keeps warm, and memory becomes the binding constraint on concurrency long before CPU does. Stateless requests, by contrast, spin a renderer up, use it, and release it, so peak memory tracks in-flight requests rather than the count of idle-but-open sessions.

Failure modes when state goes stale

Server-held state has a shelf life. Sessions time out, cookies expire, the site rotates a CSRF token, or a node restart silently drops the context. A stateful flow that assumed "I'm still logged in from step two" can fail unpredictably at step five, and the failure is hard to retry because the state that produced it is gone. You end up writing session-revival logic, heartbeats, and re-auth fallbacks — code that exists only to manage state you chose to hold.

The stateless alternative

Passing headers and cookies per request

A stateless model moves the state out of the server and into the request. Instead of the server remembering your session, you carry it: each call includes the headers and cookies the target needs. There is no warm context to route back to, because the request is complete in itself. The state still exists — it just lives in your client, where you already control it, rather than in an opaque server-side jar you cannot inspect.

Idempotent, retry-safe calls

Because a stateless request depends on nothing but its own inputs, it is idempotent and retry-safe by construction. If a call times out or a node dies, you simply send the identical request again — to any node — and get the same result, with no risk of corrupting a half-built session. This is what makes stateless scraping trivial to scale horizontally: add identical, affinity-free workers and throughput rises linearly. We cover that scaling story in depth in why a stateless request model beats sessions.

fastCRW is stateless per request

fastCRW is stateless per request by design. Each /v1/scrape call is self-contained: you hand it a URL, formats, a renderer choice, and any headers or cookies, and it returns clean content without retaining anything about you afterward. The one stateful exception is /v1/crawl, which is async and returns a job ID you poll — but that job state is a tracked background task, not a live interactive browser you drive call-by-call. Architecturally this is why fastCRW ships as a single static Rust binary (one container, no Redis, no Node sidecar) rather than the multi-service stack a session store would demand; see the fastCRW architecture overview for the full picture.

Handling auth-gated content without sessions

Forwarding cookies and bearer tokens

The common worry about stateless scraping is "how do I reach pages behind a login without a session?" The answer for a large class of sites: forward the credentials per request. If you already hold a valid session cookie or a bearer token — obtained once, out of band, or copied from your browser's dev tools — you attach it to the scrape request's headers. The target sees an authenticated request and serves the gated content, and fastCRW never has to hold a login state to make that work. This covers most token- and cookie-authenticated APIs, dashboards, and member pages.

What you can and cannot do statelessly

Stateless credential forwarding works when the gate is a header the server checks: a session cookie, an Authorization bearer token, an API key. It does not work when the gate requires live, in-browser steps you cannot replay as a single request — a login form that runs JavaScript to derive a one-time token, a CAPTCHA, a multi-page wizard, or any flow where step N's output depends on step N-1's live DOM. For those, you either perform the interactive login yourself and forward the resulting cookie, or you reach for a stateful session tool. Be honest with yourself about which category your target falls into before you build.

Where fastCRW's stateless model meets hard targets

Built-in anti-bot and proxy rotation

fastCRW ships anti-bot handling in the open core: 12-signal block detection, user-agent rotation, stealth fingerprints, and proxy rotation up to a residential-proxy egress tier for the hardest targets, all inside the same single binary rather than a bolt-on managed product. It's part of why fastCRW posts the highest truth-recall of the three tools in the canonical 3-way benchmark, 63.74% of 819 labeled URLs (diagnose_3way.py, 2026-05-08), with a fast-mode p90 of 4,348 ms that is the lowest of the three. See our anti-bot and proxies overview for how the escalation ladder works.

Matching the model to the job

For targets reachable with forwarded credentials and per-request scrapes — which is the majority of scraping and the entirety of high-throughput crawling — the stateless model is simpler, cheaper to operate, and easier to scale than holding server-side sessions. Match the model to the job, not the marketing.

Sources

Scrape benchmark of record — diagnose_3way.py, Firecrawl public dataset, 819 labeled URLs, 2026-05-08: truth-recall 63.74%, p50 1914 ms, fast-mode p90 4348 ms (lowest of three).
fastCRW open-core repo and endpoint table: github.com/us/crw.
Firecrawl /interact persistent-session behavior: docs.firecrawl.dev (verified 2026-05-18).