How do I schedule a web scraper with the bash + cron pattern?

Write a bash script that takes a flock lock (so runs never overlap), wraps the scrape call in a bounded retry loop with exponential backoff, logs every run with a timestamp, and exits non-zero on real failure so cron's MAILTO or your webhook alerts. The scrape itself is a single curl POST to a Firecrawl-compatible /v1/scrape. Add a crontab line like '0 3 * * * /path/to/scrape.sh' to run it nightly. This is the DIY pattern; for hosted recurring crawls use fastCRW's managed scheduling feature instead.

How do I stop overlapping cron scrape jobs with flock?

Open a lock file on a free file descriptor and call flock with -n at the top of the script: 'exec 9>/tmp/job.lock; flock -n 9 || exit 0'. The -n flag fails immediately if another instance holds the lock, so a job that occasionally runs long never starts a second copy that double-writes output or hammers the target. Exiting 0 on a held lock means a skipped run does not trigger failure alerts; exit non-zero instead if you want to be told about overlap.

How do I add retries and alerting to a bash scraping cron job?

Wrap the curl call (use --fail so HTTP errors return non-zero) in a bounded retry function: try up to N times with exponential backoff (2s, 4s, 8s), and return non-zero if all attempts fail. Let that failure propagate to the script's exit code. For alerting, cron emails stdout/stderr to MAILTO on non-zero exit, but for anything beyond one host pipe failures to a webhook (Slack or a healthcheck service) since unread cron mail is the same as no alert.

Should I use the DIY cron pattern or fastCRW's managed scheduling?

Use the DIY bash + cron pattern when you want data locality and the lowest cost: self-hosting the AGPL-3.0 engine is $0 per 1,000 scrapes (you pay only your server), and nothing leaves your box. Use managed scheduling when the operational tax outweighs the savings — recurring jobs across many hosts, hosted retries, run-history dashboards, and not owning the VPS. DIY wins on cost and privacy; managed wins on operational burden. The engine itself has no built-in scheduler, so in the DIY route cron is the scheduler.

Can I run a scheduled scraper on a $5 VPS?

Yes. The fastCRW engine is a single ~8 MB static binary needing 1 container (versus Firecrawl's ~2-3 GB across 5 containers, per README structural facts), so it fits a $5 VPS. Point your cron script at the local engine (e.g. http://localhost:3002/v1/scrape) and the entire pipeline stays on the box. One caveat for scheduling: In fast mode, fastCRW's p90 scrape latency is 4348 ms — the lowest of the three tools tested (Crawl4AI 4754 ms, Firecrawl 6937 ms, diagnose_3way.py, 2026-05-08) — so give each request a timeout matched to the p90 and stagger job start times rather than launching everything on the hour.

Bash Web Scraping with Cron: The CLI + Schedule Pattern

By the fastCRW team · Footprint/cost facts verified 2026-05-18 · Verify independently before relying on any figure.

Bash web scraping with cron: the DIY pattern, not the managed feature

If you want bash web scraping on a cron schedule, this post is the DIY shell pattern: a small script driven by cron, hardened with flock locking, retries, and logging, calling a Firecrawl-compatible scrape API. It is deliberately not a walkthrough of fastCRW's managed scheduling feature — for the product feature (hosted recurring crawls, no server to babysit) see scheduled crawls with cron in fastCRW. Read this when you want the scheduler to live on your own box, in your own crontab, with the output landing in your own filesystem.

The appeal of the DIY route is control and cost. cron is already on every Linux host, it has no dependencies, and the engine you call can be a single self-hosted binary — so a nightly scrape job can run end-to-end on a $5 VPS with nothing leaving your network. The catch is that most cron-scraper tutorials stop at 0 3 * * * curl ... > out.json and skip the three things that decide whether the job is still working in three months: overlap prevention, retries, and logging. We will not skip them.

Why cron is still the simplest scheduler

cron is boring in the way infrastructure should be boring. There is no daemon you wrote, no queue to drain, no state machine to debug at 2 a.m. A crontab line is a contract: this command, on this schedule, as this user. For a handful of scrape jobs — a price page nightly, a sitemap weekly, a docs site every six hours — cron is almost always the right amount of machinery. You reach for a real job runner (systemd timers, a queue, the managed feature) when you need fan-out across hosts, central observability, or per-job retry semantics that outgrow a shell script.

Anatomy of a reliable scrape cron job

A production-safe cron scraper is four concerns stacked in one script: a lock so two runs never overlap, a retry loop so a transient failure does not lose the run, logging so you can see what happened, and a non-zero exit so cron's MAILTO (or your alerting) fires on failure. The HTTP call itself — a single curl POST — is the smallest part. Everything around it is what keeps the job from silently rotting.

Making the cron job production-safe

Overlap prevention and idempotency with flock

The most common cron-scraper bug is overlap: a job scheduled every 15 minutes that occasionally takes 20 minutes, so two copies run at once, double-write the output, and hammer the target. flock fixes this with one line — it takes an exclusive lock on a file and refuses to start a second instance.

#!/usr/bin/env bash set -euo pipefail LOCK=/tmp/scrape-prices.lock # -n: fail immediately if another run holds the lock (no queueing) exec 9>"$LOCK" flock -n 9 || { echo "another run is active, exiting"; exit 0; } # ... scrape work runs here, guaranteed single-instance ...

Exiting 0 when the lock is held is deliberate: a skipped run because the previous one is still going is not a failure, so you do not want it to trigger alerts. If you would rather alert on overlap, exit non-zero instead. The lock is also your idempotency guard — write output to a temp file and mv it into place at the end, so a crash mid-run never leaves a half-written file that the next consumer reads.

Logging, exit codes, and alerting on failure

cron will email the job's stdout/stderr to MAILTO if it is set, but only if the command produces output and exits non-zero. Make that work for you: log every run with a timestamp, and let the script exit non-zero on real failure so the email actually fires.

LOG=/var/log/scrape/prices.log exec >> "$LOG" 2>&1 # all output goes to the log echo "[$(date -Is)] run start" # ... work ... echo "[$(date -Is)] run ok"

Rotate that log with logrotate so it does not grow unbounded. If you run more than a couple of jobs, pipe failures to a webhook (Slack, a healthcheck service) instead of relying on local mail — a cron email no one reads is the same as no alert at all.

Retries and backoff inside the script

Networks blip. A single failed request should not lose the whole run, so wrap the call in a bounded retry loop with exponential backoff. Bounded is the key word — infinite retries turn a dead target into a hot loop.

retry() { local max=4 delay=2 attempt=1 until "$@"; do if (( attempt >= max )); then echo "[$(date -Is)] giving up after $attempt attempts" >&2 return 1 fi echo "[$(date -Is)] attempt $attempt failed, retrying in ${delay}s" >&2 sleep "$delay" (( attempt++, delay *= 2 )) done }

Now retry curl --fail ... will try up to four times with 2s, 4s, 8s backoff and return non-zero only if all attempts fail — which then propagates to the script's exit code and your alerting.

Driving a Firecrawl-compatible API from cron

The whole point of calling a scrape API from your cron script — rather than parsing HTML in bash — is that the hard parts (JavaScript rendering, markdown conversion, structured extraction) happen server-side and you get back clean, LLM-ready output. fastCRW exposes a Firecrawl-compatible REST surface, so the same curl works against the managed cloud or a self-hosted binary by swapping the base URL.

A curl POST to /v1/scrape in the cron script

A single-page scrape is one POST. With --fail so curl returns non-zero on an HTTP error (which your retry loop and exit code depend on):

scrape() { curl --fail --silent --show-error \ -X POST "${CRW_BASE:-https://api.fastcrw.com}/v1/scrape" \ -H "Authorization: Bearer $CRW_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url":"https://example.com/prices","formats":["markdown"]}' \ -o "$TMP_OUT" } retry scrape mv "$TMP_OUT" /data/prices/$(date +%F).json

A plain markdown scrape is a 1-credit operation. If you pass formats:["json"] with a jsonSchema for structured extraction, that is the 1-credit scrape plus the LLM token cost, billed as usage-metered LLM credits that scale with page size and token usage — budget accordingly when a job runs hundreds of times a day.

Whole-site runs with /v1/crawl and a job ID

For a whole-site pass, a single scrape is the wrong tool — use /v1/crawl, which is asynchronous: the POST returns a job ID, and you poll GET /v1/crawl/:id until it completes. In a cron script that means submit, then loop on status with a sleep:

JOB=$(curl --fail -s -X POST "$CRW_BASE/v1/crawl" \ -H "Authorization: Bearer $CRW_API_KEY" -H "Content-Type: application/json" \ -d '{"url":"https://example.com","maxPages":200}' | jq -r '.id') while :; do STATUS=$(curl --fail -s "$CRW_BASE/v1/crawl/$JOB" \ -H "Authorization: Bearer $CRW_API_KEY" | jq -r '.status') [[ "$STATUS" == "completed" ]] && break [[ "$STATUS" == "failed" ]] && { echo "crawl failed" >&2; exit 1; } sleep 10 done

maxPages caps at 1000 and maxDepth at 10. If a job stalls, DELETE /v1/crawl/:id cancels it — wire that into a trap so a killed cron run does not leave a crawl running server-side.

Output rotation, dedupe, and filtering with jq

Once the data lands, jq does the shaping in-pipeline: pull the fields you care about, drop duplicates, and write a dated file. Rotate output by date so you keep history without one file growing forever, and dedupe against yesterday's run if you only want changes:

jq -r '.data[] | {url, title, price}' "$TMP_OUT" \ | sort -u > /data/prices/$(date +%F).ndjson

Self-hosting for fully local scheduled runs

Running the single binary on a small VPS

The structural fact that makes the DIY pattern cheap: the fastCRW engine is a single ~8 MB static binary needing 1 container, versus Firecrawl's ~2–3 GB across 5 containers (README structural facts, not a benchmark claim). That footprint fits comfortably on a $5 VPS, so your cron script can target http://localhost:3002/v1/scrape and nothing — not the target URLs, not the scraped content — ever leaves the box. See running fastCRW on a $5 VPS and self-hosting with Docker Compose for the setup.

Cost of a whole-site pass when you self-host

When you self-host the AGPL-3.0 engine, the per-scrape price is $0 per 1,000 scrapes — you pay only for the server. A nightly crawl of a few hundred pages on a $5 VPS therefore costs you the $5/month and nothing per request, which is the whole argument for the DIY-on-your-own-iron route over a metered API for high-frequency recurring jobs.

When managed scheduling beats DIY cron

DIY cron stops being the right call when the operational tax exceeds the savings. If you need recurring jobs across many hosts, a dashboard of run history, hosted retries, and someone else owning uptime, the managed scheduling feature removes the crontab, the lock files, the log rotation, and the VPS itself. The honest framing: DIY cron wins on cost and data locality; managed wins on operational burden. Compare against live pricing for your volume before deciding.

Scaling the cron pattern

Sharding across hosts

One crontab on one host has a ceiling. To scale, shard the URL list across N hosts — host i handles URLs where hash(url) % N == i — so each box runs the same script over its slice. This keeps the per-host script unchanged and turns scaling into a configuration problem rather than a rewrite.

Staggering jobs to smooth the latency tail

This is where an honest benchmark fact changes how you schedule. On Firecrawl's public dataset (819 labeled URLs, diagnose_3way.py, 2026-05-08), fastCRW's median scrape is fast — p50 1914 ms, beating Firecrawl's 2305 ms — and in fast mode its p90 is 4348 ms, the lowest of the three tools tested (Crawl4AI 4754 ms, Firecrawl 6937 ms). The chrome-stealth fallback that recovers the 34 URLs others miss is the same mechanism that keeps the tail competitive. For a cron pattern, the lesson is concrete: do not start every job at 0 * * * *. Stagger start minutes and give each curl a timeout matched to the p90, and let a fraction of pages take their full recovery time. A job that assumes every scrape returns in two seconds will spuriously fail on exactly the pages fastCRW is good at recovering.

Stateless per request, cron is the scheduler

Two things to state plainly. First, the engine has no built-in scheduler — cron is the scheduler in this pattern; the engine just answers requests. Second, the engine is stateless per request: there is no session that persists between cron runs, so anything you want to remember (last-seen prices, a dedupe set) lives in your filesystem, not the engine. Two capabilities worth knowing for scheduled jobs: screenshot output is supported (formats:["screenshot"] returns a base64 PNG via the Chrome/CDP renderer), and /v1/extract batches structured extraction across up to 50 URLs in a single call rather than looping /v1/scrape.

Sources

fastCRW canonical facts : github.com/us/crw README structural footprint + endpoint table
Scrape benchmark of record (p50/p90, 819 labeled URLs, diagnose_3way.py, 2026-05-08): bench/server-runs/RESULT_3WAY_1000_FULL.md
flock and cron behavior: man7.org flock(1) · man7.org crontab(5)