By the fastCRW team · Ruby/Rails tutorial & analysis · Last reviewed 2026-01-01
Disclosure: fastCRW is a scraping engine with an HTTP API; this guide is Rails-pragmatic first and shows the engine path where it removes a real Rails operational problem.
The Ruby scraping reality: it's almost always inside a Rails app
People scraping in Ruby are rarely writing a standalone crawler — they're adding "ingest this external page" to an existing Rails application. That context changes everything, because the hard part isn't parsing HTML in Ruby (Nokogiri is excellent and has been for over a decade), it's that scraping is a slow, failure-prone, externally-dependent operation living inside an app whose request/response cycle and background-job queue were not designed to absorb it. This guide is about that integration problem, which is the actual Ruby-scraping problem.
The Ruby stack (this part is easy)
nokogiri— the parsing workhorse: libxml2-backed, fast, full CSS and XPath. Battle-tested; you will not out-engineer it.http(HTTP.rb) orfaraday— clean HTTP clients with timeouts and middleware. Prefer explicit timeouts always (default-infinite timeouts are the classic Rails scraping outage).ferrum/cuprite— drive headless Chrome from Ruby for JS-rendered pages. Same caveat as every ecosystem: this drags a Chrome dependency into your deployment.
Minimal Nokogiri example
require "http"
require "nokogiri"
html = HTTP.timeout(connect: 3, read: 8).get("https://example.com/").to_s
doc = Nokogiri::HTML(html)
title = doc.at_css("article h1")&.text
puts title
The Ruby-specific trap: scraping inside the request cycle, then inside the job queue
The first mistake is scraping in a controller action — a 6-second external fetch pins a Puma/Unicorn worker, and under any load your Rails app falls over. Every Ruby dev learns to move it to a background job. The second, subtler trap is what happens next: you put scraping in Sidekiq/ActiveJob, and now scraping reliability is your queue's reliability problem. Concretely:
- Retries amplify load. Sidekiq's default retry will re-run a failed scrape up to ~25 times with backoff. A site that's blocking you now gets hammered 25× per URL, burning your IP reputation and your worker capacity.
- Long scrapes starve the queue. A pool of slow scrape jobs (each waiting on a hostile remote server) occupies workers that your transactional jobs (emails, webhooks) need. Scraping latency becomes app-wide latency.
- Memory bloat. Nokogiri on large documents, multiplied across concurrent Sidekiq threads, plus a Chrome process if you added
ferrum— Rails memory footprint balloons and you're tuning Sidekiq concurrency to survive scraping, not your actual workload. - Failure handling leaks into your domain. Anti-bot blocks, partial pages, and timeouts now need handling in job code that sits next to your business logic, coupling fragile external behavior to your app's core.
This is the Ruby-scraping insight nobody tells you: the parsing is trivial; making scraping a well-behaved tenant of a Rails background-job system is the real, recurring engineering cost.
The Rails-clean architecture: scraping as an external service, not a Sidekiq tenant
The fix that keeps your Rails app healthy is to stop treating scraping as Ruby code and start treating it as a call to a service that specializes in it. Your Sidekiq job becomes a thin, fast, predictable HTTP call to a scraping engine that already owns retries, anti-bot, rendering, and politeness — so none of that lives in your queue:
class IngestPageJob
include Sidekiq::Job
sidekiq_options retry: 3 # small, sane — the ENGINE owns scrape retries
def perform(url)
res = HTTP.auth("Bearer #{ENV['CRW_API_KEY']}")
.timeout(connect: 3, read: 20)
.post("#{ENV['CRW_BASE_URL']}/v1/scrape",
json: { url: url, formats: ["markdown"] })
Document.upsert_from(url, res.parse["data"]["markdown"])
end
end
What this fixes in Rails terms: the job is now a bounded, predictable HTTP call (easy to set timeouts and a small retry on), Chrome is no longer in your deployment, anti-bot maintenance is out of your codebase, and the queue-starvation/retry-amplification pathologies disappear because the slow, flaky part is behind a service built for it. Because fastCRW is open-core (AGPL-3.0) you can run the engine as a container next to your Rails app in development and data-sensitive environments, then point CRW_BASE_URL at the Managed Cloud in production — no Ruby code change, just an env var, which is exactly how Rails apps already manage configuration.
The Ruby-specific second trap: idempotency and the at-least-once queue
There's a deeper Sidekiq trap beyond retry-amplification, and it's specific to how Ruby teams wire scraping into a database-backed app. Sidekiq (and most Ruby job systems) are at-least-once: a job can run twice — worker killed mid-execution, network blip on the ack, a deploy restarting workers. If your scrape job does "fetch page, then Document.create!", a double-run creates duplicate rows or, worse, a partially-applied second run corrupts derived data (search indexes, counters, embeddings). The discipline Ruby scraping demands is end-to-end idempotency: upsert keyed by canonical URL, content-hash guards so an unchanged page is a no-op, and making any side effect (reindex, webhook, embed) safe to repeat. This is real engineering effort that has nothing to do with HTML and everything to do with Rails data integrity. Moving the scrape itself behind a clean HTTP call doesn't remove the idempotency requirement, but it shrinks the job to "one deterministic fetch + one upsert," which is dramatically easier to make idempotent than "fetch + retry + render + parse + multi-table write" all in one fragile job.
Connection pools, threads, and the Sidekiq concurrency math
A Ruby-specific operational subtlety: Sidekiq runs jobs on threads within a process, and every scraping job that also touches the database holds an ActiveRecord connection while it waits on a slow external server. If Sidekiq concurrency is 25 and each scrape job blocks for 6 seconds on a remote fetch while holding a DB connection, you can exhaust your connection pool and stall unrelated jobs and web requests that needed those connections. The fix when scraping in-process is fiddly (release the connection around the HTTP call, tune pool size, isolate scraping to its own queue and process). When the scrape is a thin call to an external engine, the blocking window shrinks and you can deliberately not hold a DB connection across it — the architecture makes the connection-pool math tractable instead of a tuning exercise you revisit every time traffic grows.
When hand-rolled Ruby scraping is still right
- A handful of known, friendly, static pages (a partner's price list, a public dataset) → Nokogiri in a background job is perfectly fine; don't add infrastructure.
- One-off rake-task scrapes / data migrations → just write the Ruby; it's the fastest path.
- You explicitly want the scraped HTML parsing coupled to ActiveRecord models and the volume is low → reasonable.
Reach for the engine when scraping becomes recurring, the targets get hostile, you need JS rendering without Chrome in your Rails deployment, or you notice you're tuning Sidekiq around scraping instead of your real jobs.
The Rails-specific transaction trap
One more Rails-idiomatic mistake worth naming: performing a scrape inside a database transaction. It happens innocently — a service object opens a transaction, does some writes, calls out to scrape "while we're here," then commits. Now an external HTTP fetch of arbitrary, possibly slow, possibly hanging duration is holding an open database transaction and its row locks. A handful of these under load escalates into lock contention and connection-pool exhaustion that manifests as site-wide slowness with a cause that's invisible unless you know to look for "transaction open across an HTTP call." The Rails discipline is absolute: never perform network I/O inside a transaction; scrape first (or via a separate job), then open the shortest possible transaction to persist the result. Shrinking the scrape to a single bounded external call makes this rule trivial to follow, because the call is fast, predictable, and obviously belongs outside any transaction — whereas a sprawling in-process fetch-render-parse is exactly the kind of thing that tempts a developer to "just wrap it" and quietly destabilize the database.
Bottom line
Web scraping in Ruby is not a parsing problem — Nokogiri solved that long ago — it's a "don't let scraping destabilize your Rails app and its job queue" problem. The traps are scraping in the request cycle, then retry-amplification and queue-starvation once it's in Sidekiq. The Rails-clean pattern is to make scraping a thin, bounded HTTP call to an engine that owns the flaky parts, configured by env var so the same code runs against a self-hosted engine or the Managed Cloud. fastCRW's open-core single-binary engine fits that pattern without trading Rails simplicity for vendor lock-in.
Try it from Rails
docker compose up # run the engine beside your Rails app, AGPL-3.0Managed Cloud, same API: one-time lifetime 500 free credits, no card. fastcrw.com · GitHub
Related: Anti-bot & proxies overview · Web scraping in PHP · LLM-ready markdown extraction