What is the best library for web scraping in Go?

colly for crawling/scheduling and goquery for DOM extraction is the de facto stack. chromedp drives headless Chrome for JS-rendered pages but adds a Chrome dependency that breaks Go's single-static-binary advantage.

What's the hardest part of scraping in Go?

Not goroutines — backpressure. Unbounded goroutines get you IP-banned and exhaust local resources. Bounded concurrency with a worker pool, semaphore, and context cancellation is the real production skill. Anti-bot is the second wall and is not solvable with Go expertise alone.

How do I keep a static binary while scraping JS-heavy sites in Go?

Don't embed Chrome in your binary. Call a scraping engine over HTTP for fetch/render/extract so your Go service stays a pure static binary. fastCRW is a single-binary engine with a Firecrawl-compatible API you can self-host as a sidecar or use via its Cloud with the same code.

Web Scraping in Go (2026): Goroutines, Backpressure, and When to Stop Building It Yourself

By the fastCRW team · Go tutorial & analysis · Last reviewed 2026-01-01

Disclosure: fastCRW is a Rust scraping engine with an HTTP API; this guide shows the Go-native path first and honestly, then where calling an engine beats hand-rolling.

Why Go is a natural scraping language — up to a point

Go's appeal for scraping is structural and real: goroutines make massive I/O concurrency trivial, the standard net/http is production-grade, and you ship a single static binary — the same deployment ethos that makes a compiled scraping engine attractive. For fetching and parsing well-behaved sites, idiomatic Go is excellent and you should just write it. This guide is about the Go-native path and the specific wall it hits, because pretending the wall doesn't exist is how teams waste a quarter.

The Go-native stack

The mature toolkit:

colly — the de facto crawling framework: request scheduling, callbacks, rate limiting, cookie handling. The right starting point.
goquery — jQuery-style DOM querying over net/html. Comfortable selector ergonomics for extraction.
chromedp — drives headless Chrome via the DevTools Protocol for JS-rendered pages. This is where Go's "single static binary" promise ends — you now depend on a Chrome install, the heaviest thing in your deployment.

Minimal colly + goquery example

c := colly.NewCollector(
    colly.AllowedDomains("example.com"),
    colly.MaxDepth(2),
)
c.Limit(&colly.LimitRule{DomainGlob: "*", Parallelism: 8, Delay: 200 * time.Millisecond})

c.OnHTML("article", func(e *colly.HTMLElement) {
    title := e.ChildText("h1")
    body := e.ChildText("p")
    fmt.Println(title, len(body))
})
c.Visit("https://example.com/")

The Go-specific lesson: concurrency is easy, backpressure is the actual skill

Every Go scraping tutorial shows you go fetch(url) in a loop. That is the trap. Unbounded goroutines will (a) get you rate-limited or IP-banned within minutes because you hammered the target, and (b) exhaust file descriptors and memory on your side. The Go-specific competence is bounded concurrency with backpressure: a worker pool sized to a politeness budget, a semaphore channel, context-based cancellation so a slow host can't pin a worker forever.

sem := make(chan struct{}, 16)        // max 16 in flight
for _, u := range urls {
    sem <- struct{}{}
    go func(u string) {
        defer func() { <-sem }()
        ctx, cancel := context.WithTimeout(context.Background(), 8*time.Second)
        defer cancel()
        scrape(ctx, u)
    }(u)
}

This — not the goroutine itself — is the part that takes a week to get right in production, and it's identical work no matter how good your parser is. It's also the first reason teams reach for an engine: a scraping service has already solved admission control and per-host politeness so your Go code doesn't reimplement it.

The wall: anti-bot is not a Go problem you can out-code

Here's the honest part. colly + goquery handles a large fraction of the web. Then you hit Cloudflare/WAF-fronted targets and discover the problem isn't your Go — it's TLS fingerprinting, JS challenge pages, and IP reputation (covered in our anti-bot overview). Go's default http.Transport has a recognizable TLS fingerprint; you can fight it with uTLS, then you need residential proxies, then a stealthed chromedp, then challenge handling. None of this is Go expertise — it's a separate, never-finished specialty, and maintaining it in your Go service is pure cost that doesn't make your product better.

Calling a scraping engine from Go (keeping the static-binary ethos)

The clean architecture: keep your Go service for orchestration and business logic, hand the fetch/render/anti-bot/extract problem to an engine over HTTP. This preserves everything Go-idiomatic about your codebase (still a static binary, still goroutine-concurrent) while deleting the anti-bot maintenance burden. fastCRW is a fitting target because it's itself a single static binary with a Firecrawl-compatible HTTP API — the call is just net/http:

type scrapeReq struct {
    URL     string   `json:"url"`
    Formats []string `json:"formats"`
}
body, _ := json.Marshal(scrapeReq{URL: target, Formats: []string{"markdown"}})
req, _ := http.NewRequestWithContext(ctx, "POST",
    baseURL+"/v1/scrape", bytes.NewReader(body))
req.Header.Set("Authorization", "Bearer "+apiKey)
resp, err := http.DefaultClient.Do(req)
// resp.Body -> clean markdown, no Chrome dependency in YOUR binary

Note what disappeared: no chromedp, so your Go deployment is a true static binary again with no Chrome dependency; no uTLS/proxy plumbing; goroutine concurrency now just fans out HTTP calls to an engine that already handles per-host politeness. And because the engine is open-core (AGPL-3.0), you can run it as a sidecar in your own infra during development or for data-sensitive workloads, then point the same Go code at the Managed Cloud by changing one base URL — no code change.

The Go-idiomatic integration: contexts, errgroup, and circuit breaking

Calling an engine from Go is not just "make an HTTP request" — done idiomatically it's where Go's concurrency primitives actually shine, because now you're orchestrating fan-out over a reliable service instead of fighting flaky targets directly. The production pattern uses errgroup for bounded fan-out with first-error cancellation, propagates a request-scoped context.Context so a cancelled upstream request tears down all in-flight scrapes, and wraps the engine call in a circuit breaker so a transient engine hiccup degrades gracefully instead of stampeding. This is the part Go is genuinely excellent at — it just wasn't the fetch-and-parse part. Crucially, when the engine owns per-host politeness and anti-bot, your Go concurrency code no longer needs per-domain rate limiters, retry backoff state machines, or proxy-pool bookkeeping; it collapses to "fan out N calls, respect context, surface errors." That deletion of accidental complexity is the real Go win of the engine architecture, not merely the deleted Chrome dependency.

Error handling: distinguishing the four failure classes

A robust Go scraper must distinguish four failure classes, and conflating them is the most common production bug. Transport errors (DNS, connection refused, TLS) are retryable with backoff. Target errors (the site returned 404/410) are permanent and must not be retried — retrying them is how you waste budget and hammer sites. Anti-bot blocks (403/429/challenge page) are retryable only with escalation (different IP, render) — naive retry makes them worse. Extraction-empty (200 OK but no meaningful content, typically a JS shell) needs a render escalation, not a retry of the same fetch. Hand-rolled Go scrapers routinely collapse all four into "if err != nil { retry }", which is precisely how you get IP-banned and produce silent corpus gaps. An engine that classifies and handles these internally means your Go code sees a clean two-way split — usable content, or a typed terminal error — which is the only sane contract to build orchestration on.

When to stay Go-native vs. call an engine

Stay Go-native: targets are non-hostile, volume is modest, the scrape logic is simple, and you enjoy owning it. colly + goquery is genuinely great here.
Call an engine: you've hit the anti-bot wall, you need JS rendering without a Chrome dependency in your binary, you want one integration that works self-hosted and managed, or the maintenance is eating sprint time that should go to your product.

Observability: the thing Go scrapers ship without and regret

A Go scraping service that lacks structured per-target metrics is operating blind, and this bites specifically because scraping fails partially and silently rather than loudly. The Go-idiomatic fix is to instrument from the start: a Prometheus counter per outcome class (success, transport-fail, blocked, empty-extract) labeled by host, a histogram of per-request latency, and structured logs that capture the final URL after redirects and the decision (static vs rendered). Without this, "the corpus has gaps" is an unfalsifiable complaint with no telemetry to localize it; with it, you can see that one host's success rate fell off a cliff last Tuesday and act before the RAG answers degrade. The relevant point for the build-vs-engine decision: an engine that already exposes these outcome metrics gives you the observability for free, whereas a hand-rolled Go scraper means you also own building, maintaining, and dashboarding the telemetry — more undifferentiated work that has nothing to do with your product but everything to do with whether you can trust your data.

Bottom line

Web scraping in Go is excellent for the fetch-and-parse majority — goroutines plus colly/goquery, with bounded-concurrency backpressure as the real skill. The wall is anti-bot, and it is not a Go problem you can out-code; it's a separate specialty. The pragmatic Go architecture keeps your idiomatic static-binary service and offloads fetch/render/anti-bot to an engine over HTTP — and choosing an open-core engine like fastCRW means that offload doesn't trade one lock-in for another.

Try it from Go

docker compose up   # run the engine locally as a sidecar, AGPL-3.0

Managed Cloud, same API: one-time lifetime 500 free credits, no card. fastcrw.com · GitHub