cURL Web Scraping: How to Scrape Pages With cURL

By the fastCRW team · Tutorial verified 2026-05-29 · Verify independently before relying on any figure.

cURL web scraping is the fastest way to pull a page from the command line: one binary, no install, and the raw bytes the server sent in under a second. This tutorial teaches cURL scraping honestly — requests, headers, cookies, redirects, parsing — and then shows the exact wall where cURL stops working (JavaScript rendering, anti-bot fingerprinting, no markdown conversion). When you hit that wall, you do not have to abandon the command line: a single POST /v1/scrape call to a Firecrawl-compatible API like fastCRW is still just a curl command, and it returns clean markdown instead of a megabyte of script tags.

Disclosure: we build fastCRW, an open-core (AGPL-3.0) scraping engine. The cURL half of this guide is vendor-neutral and true regardless of what you scrape with; the "when to switch" half is interested, so we have kept it testable and stated the gaps plainly.

cURL web scraping basics: fetching pages

cURL (Client URL) is a command-line HTTP client, and for cURL web scraping it does exactly one thing: it sends a request and prints the response body — nothing parsed, nothing rendered. For static pages, server-rendered HTML, and JSON APIs, that is often all you need. Confirm it is installed:

curl --version

Fetch a page and print it to the terminal, or save it to a file:

# Print to stdout
curl https://example.com

# Save to a file
curl https://example.com -o page.html

# Headers only (HEAD request) — check status and content-type cheaply
curl -I https://example.com

The flags you will reach for constantly:

A practical default for scraping is curl -sSL --compressed: quiet, follows redirects, surfaces real errors, and asks for compressed transfer so you are not pulling uncompressed HTML over the wire.

Sending headers, cookies, and handling redirects

The single biggest reason a cURL request gets a different response than your browser is the headers. cURL identifies itself as curl/8.x by default, and plenty of servers either block that or serve a stripped-down page. Override the User-Agent and add the headers a real browser sends:

curl -sSL --compressed \
  -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" \
  -H "Accept-Language: en-US,en;q=0.9" \
  -H "Accept: text/html,application/xhtml+xml" \
  https://example.com -o page.html

Cookies and sessions. cURL's cookie engine is off until you turn it on. Use -c to write the cookies a server sets, then -b to send them back on the next request. Together they keep a session alive across calls:

# 1) Land on the site, capture Set-Cookie into cookies.txt
curl -sSL -c cookies.txt -o /dev/null https://example.com

# 2) Reuse those cookies on a protected page (and keep updating the jar)
curl -sSL -b cookies.txt -c cookies.txt https://example.com/account -o account.html

Cookies are stored in Netscape format, so you can export a logged-in browser session and pass it straight to cURL — useful for one-off authenticated pulls, brittle as a long-term strategy because tokens expire.

Redirects. Without -L, cURL prints the 3xx response and stops. With -L it follows the chain; cap it with --max-redirs N if you want to detect redirect loops instead of silently following them.

POST and pagination. Many "pages" are really JSON endpoints. Send a body with -d and walk an offset/cursor in a shell loop:

for page in $(seq 1 5); do
  curl -sSL --compressed \
    "https://api.example.com/items?page=$page&limit=50" \
    -o "items-$page.json"
done

This is genuinely the right tool for paginated JSON APIs. It only falls apart when the "API" is actually a JavaScript app rendering HTML in the browser — which is the wall we hit below.

Parsing and cleaning the raw HTML

cURL gives you a string. Turning that string into fields is entirely on you. For quick checks you can pipe into command-line tools:

# Extract the '

# JSON APIs: pipe straight into jq
curl -sSL https://api.example.com/items | jq '.data[].name'

For anything structured you graduate to a real parser — BeautifulSoup or lxml in Python, Cheerio in Node — and write a CSS/XPath selector for every field you want. That works, but you now own three liabilities: the selectors break whenever the site changes its markup, you maintain extraction logic per field, and you are still only parsing whatever HTML the server sent on the first request. If the content arrives later via JavaScript, no parser can find what was never in the bytes. For the general shape of HTML-to-text and a markdown-first alternative, see turning any website into clean markdown.

Where cURL breaks: JavaScript and anti-bot

cURL has no JavaScript engine and no HTML parser. On a site built with React, Next.js, Vue, or any client-rendered framework, the server sends a loader and the browser executes scripts to produce the visible content. cURL gets the loader. Run it against a modern SPA and the saved file is mostly this:

<div id="__next"></div>
<script src="/_next/static/chunks/framework-1c17a9c2.js" defer></script>
<script src="/_next/static/chunks/main-app-a9e61de8.js" defer></script>

The prices, headlines, and product data you actually wanted are nowhere in the response — they only exist after the scripts run. Two stopgaps exist when cURL is your only tool, and both are fragile:

• Hit the underlying API. Open DevTools, watch the Network tab filtered to XHR/Fetch, and find the JSON endpoint the page calls. Often you can curl that directly for clean structured data. It works until the undocumented endpoint changes shape without notice.
• "Copy as cURL." Right-click a request in DevTools and copy it as a cURL command with all cookies and headers pre-filled. Great for reproducing one authenticated request; the session tokens expire and you rebuild it each time.

Beyond rendering, three more walls show up at any real volume:

• Anti-bot and TLS fingerprinting. Sites increasingly fingerprint the TLS handshake to tell scripted clients from browsers. cURL's fingerprint is not Chrome's, so a server can block it no matter what User-Agent you set. Headers alone do not get you past this — see our overview of anti-bot defenses and proxies for what actually moves the needle.
• No retries or rate limiting. cURL fires as fast as the connection allows. When a site returns 429 or 503, there is no built-in exponential backoff — you script it yourself.
• No structured, LLM-ready output. Even when cURL gets the HTML, you still own the markdown/JSON conversion. For RAG and agents that means you are token-bloating context with tags, or maintaining a cleaner.

The full treatment of rendering strategies lives in our JavaScript web scraping guide. The short version: once a site needs a browser, raw cURL is the wrong layer to solve it at.

Calling a scraping API with curl instead

Here is the part that keeps you on the command line. A managed scrape endpoint runs a real browser server-side, applies the anti-bot path, and converts the result to clean markdown — and you call it with the same curl you already know. fastCRW exposes a Firecrawl-compatible REST surface, so the request is a plain JSON POST:

curl -sS -X POST https://api.fastcrw.com/v1/scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "url": "https://example.com/pricing",
    "formats": ["markdown"]
  }'

That returns JSON with a data.markdown field containing the rendered page as clean, structured text — no script tags, no DOM cleanup on your side, ready to drop into an LLM context window. The renderer auto-selects (chrome → lightpanda → http fallback), so JavaScript pages are handled without you choosing an engine. Because the surface is Firecrawl-compatible, the official Firecrawl SDK works against fastCRW after a single base-URL swap:

# Point an existing Firecrawl client at fastCRW — one line
from firecrawl import FirecrawlApp

app = FirecrawlApp(
    api_key="YOUR_API_KEY",
    api_url="https://api.fastcrw.com",  # the only change
)
result = app.scrape_url("https://example.com/pricing", formats=["markdown"])
print(result["markdown"][:500])

The same surface gives you /v1/crawl for whole sites, /v1/map to discover every URL, and /v1/search for web search — see the scrape endpoint deep dive for formats, JSON extraction, and error handling. One honest note on scale: there is no multi-URL batch endpoint, so for many pages you iterate /v1/scrape concurrently or run a /v1/crawl job — the same pattern your cURL --parallel loop already expresses.

cURL vs a managed scrape endpoint

Does the managed path actually get more content? On accuracy, fastCRW recorded the highest truth-recall of three tools tested — 63.74% of 819 labeled URLs — on Firecrawl's own public scrape dataset (diagnose_3way.py, 2026-05-08), ahead of Crawl4AI (59.95%) and Firecrawl (56.04%). Median latency was 1914 ms (p50), beating Firecrawl's 2305 ms. In fast mode, fastCRW's p90 is 4348 ms — the lowest of the three (Crawl4AI 4754 ms, Firecrawl 6937 ms). fastCRW also recovers 34 URLs that neither Crawl4AI nor Firecrawl reach — 70% more exclusive recoveries than the other two combined. We publish the full p50/p90 split rather than a single average — see /benchmarks for the methodology. None of these numbers describe raw cURL, which does not render JavaScript at all; they describe the layer you switch to when cURL stops returning content.

What it costs. A scrape is 1 credit on any renderer — HTTP, lightpanda, or chrome. The Free tier is 500 one-time lifetime credits — enough to test a pipeline — and the engine itself is AGPL-3.0, so self-hosting is $0 per 1,000 scrapes beyond your own server. Paid tiers are Hobby $13, Standard $69, Growth $279, and Scale $549; check live numbers on /pricing rather than trusting a figure baked into a blog post. The source is open at github.com/us/crw.

The decision in one line. Keep cURL for what it is genuinely best at — quick checks, API exploration, paginated JSON, isolating whether a bug is in your code or the server's response. Switch to a managed scrape endpoint the moment you need rendered content, clean markdown, or anti-bot at volume. And because the endpoint is itself a curl call, switching does not mean leaving the command line.

Sources

• fastCRW scrape benchmark, 3-way on Firecrawl's public 1,000-URL dataset (819 labeled), diagnose_3way.py, 2026-05-08 — truth-recall 63.74%, p50 1914 ms, p90 4348 ms (fast mode, lowest of three): /benchmarks
• fastCRW pricing, credit costs, and renderer selection: /pricing · github.com/us/crw
• cURL documentation (flags, cookies, proxies): curl.se/docs

Related: JavaScript web scraping · Website to markdown · Firecrawl scrape endpoint deep dive · Anti-bot and proxies overview