Web Scraping for Content Aggregation

Content Aggregation Needs More Than RSS

Content aggregation — pulling articles, posts, documentation, and product data from many sources into a unified corpus — sits at the foundation of search engines, AI training datasets, RAG knowledge bases, market research dashboards, and curated content products. RSS feeds handle a fraction of this problem. Web scraping handles the rest.

The gap RSS leaves behind is large:

• Partial content: most RSS feeds publish summaries or the first 250 words, not full text
• Missing sources: forums, documentation sites, e-commerce review sections, government portals, and niche blogs rarely publish RSS at all
• Stale or removed feeds: publishers quietly drop feeds without redirects, breaking RSS-dependent pipelines silently
• No structured metadata: RSS titles and pubDate fields exist, but author bios, category tags, reading time, and linked entities require scraping

A mature content aggregation pipeline uses RSS where it exists and scraping everywhere else. fastCRW closes the scraping side of that gap with a single Firecrawl-compatible REST API that returns clean markdown and structured JSON from any public URL.

RSS Feeds vs. Web Scraping: A Direct Comparison

Criterion	RSS / Atom Feed	Web Scraping (fastCRW)
Content completeness	Summaries only (most feeds)	Full body text, always
Source coverage	Only publishers who maintain feeds	Any publicly accessible URL
Metadata richness	Title, pubDate, link — nothing else	Author, tags, publish date, reading time, images — all extracted via JSON schema
Freshness control	Depends on publisher's feed update schedule	You control re-scrape cadence
Feed reliability	Feeds silently go stale, 404, or get removed	Source page exists as long as the domain is live
JavaScript-rendered content	No (feed is static XML)	Yes — Chrome and LightPanda renderers handle dynamic pages
Paywalled content	Feeds may include gated summary	Scraping behind login requires credentials (not supported by fastCRW without session cookies)
Cost	Free to poll	1 credit per page (any renderer)
Implementation effort	Feed parser library, one line	One HTTP call per URL to /v1/scrape

Verdict: Use RSS as a fast, low-cost trigger to detect new content on sources that maintain feeds. Use fastCRW scraping to fetch the full text, to cover sources without feeds, and to extract metadata not available in the feed. The two complement each other — they are not alternatives.

Where fastCRW Fits in the Aggregation Stack

Aggregation need	fastCRW endpoint	Notes
URL discovery across a domain	POST /v1/map	Returns all pages found via sitemap + crawl; 1 credit per domain
Full-text extraction	POST /v1/scrape	Clean markdown output, JS-rendered if needed; 1 credit per page
Recursive collection	POST /v1/crawl	BFS crawl up to 1,000 pages / 10 levels; 1 credit per page
Structured metadata	POST /v1/scrape + formats: ["json"]	JSON schema-driven extraction (headline, author, date, summary); 5 credits
Topic-based discovery	POST /v1/search	Search for content on a topic across the web; 1 credit per query
AI agent access	POST /mcp	Streamable HTTP MCP — connect Claude, Cursor, or any MCP client directly

The Aggregation Pipeline Architecture

A production content aggregation pipeline has seven layers. Here is the full flow:

Sources (URLs / domains)
        │
        ▼
┌────────────────────┐
│  1. MAP            │  POST /v1/map per domain → full URL list
│  (discovery)       │  Run weekly; costs 1 credit per domain
└────────┬───────────┘
         │
         ▼
┌────────────────────┐
│  2. FILTER         │  Pattern-match URLs: keep /article/, /blog/, /post/
│  (URL selection)   │  Discard nav, tag, login, search result pages
└────────┬───────────┘
         │
         ▼
┌────────────────────┐
│  3. SCRAPE         │  POST /v1/scrape per URL → markdown + metadata
│  (full-text fetch) │  Batch concurrently; 1 credit per page (any renderer)
└────────┬───────────┘
         │
         ▼
┌────────────────────┐
│  4. EXTRACT        │  formats: ["json"] + jsonSchema → headline, author,
│  (structured meta) │  date, summary, category; 5 credits per page
└────────┬───────────┘
         │
         ▼
┌────────────────────┐
│  5. DEDUPE         │  SHA-256 hash (exact dup) + embedding cosine sim
│  (uniqueness gate) │  (near-dup); skip already-seen hashes
└────────┬───────────┘
         │
         ▼
┌────────────────────┐
│  6. STORE          │  PostgreSQL / Elasticsearch / vector DB
│  (persistence)     │  With source URL, scraped_at, content hash, metadata
└────────┬───────────┘
         │
         ▼
┌────────────────────┐
│  7. RE-CRAWL       │  Cron: re-map weekly, re-scrape daily (changed URLs)
│  (freshness loop)  │  Compare hashes to detect updated content
└────────────────────┘

Each layer is independent — you can swap storage backends, add vector indexing, or insert an LLM summarization step without changing the scraping layer.

Code Examples

1. Discover all URLs on a content domain (curl)

curl -X POST https://api.fastcrw.com/v1/map \
  -H "Authorization: Bearer $FASTCRW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://techcrunch.com"
  }'

Response (truncated):

{
  "success": true,
  "urls": [
    "https://techcrunch.com/2026/06/10/article-slug-here/",
    "https://techcrunch.com/2026/06/09/another-article/",
    "..."
  ]
}

Use the returned URL list as the feed into your filter step. /v1/map traverses sitemaps and crawl paths — you get the full site graph without recursive crawling logic in your own code.

2. Full pipeline in Python

This script maps a set of domains, filters for article URLs, scrapes full text, deduplicates by content hash, and extracts structured metadata:

import requests
import hashlib
import json
from typing import Optional

FASTCRW_BASE = "https://api.fastcrw.com/v1"
FASTCRW_KEY = "your-api-key"

HEADERS = {"Authorization": f"Bearer {FASTCRW_KEY}", "Content-Type": "application/json"}

# --- Step 1: Map a domain to discover all URLs ---

def map_domain(url: str) -> list[str]:
    resp = requests.post(f"{FASTCRW_BASE}/map", headers=HEADERS, json={"url": url})
    resp.raise_for_status()
    return resp.json().get("urls", [])

# --- Step 2: Filter URLs to content pages only ---

CONTENT_PATTERNS = ["/article/", "/blog/", "/post/", "/news/", "/story/"]

def is_content_url(url: str) -> bool:
    return any(p in url for p in CONTENT_PATTERNS)

# --- Step 3: Scrape a single URL into markdown ---

def scrape_page(url: str) -> dict:
    payload = {
        "url": url,
        "formats": ["markdown"],
    }
    resp = requests.post(f"{FASTCRW_BASE}/scrape", headers=HEADERS, json=payload)
    if resp.status_code != 200:
        return {}
    return resp.json().get("data", {})

# --- Step 4: Extract structured metadata via JSON schema ---

ARTICLE_SCHEMA = {
    "type": "object",
    "properties": {
        "headline":       {"type": "string", "description": "Article headline"},
        "author":         {"type": "string", "description": "Author full name"},
        "published_date": {"type": "string", "description": "Publication date, ISO 8601"},
        "summary":        {"type": "string", "description": "One-sentence summary of the article"},
        "category":       {"type": "string", "description": "Topic category (Tech, Finance, World, etc.)"},
    },
    "required": ["headline", "published_date"],
}

def extract_metadata(url: str) -> dict:
    payload = {
        "url": url,
        "formats": ["json"],
        "jsonSchema": ARTICLE_SCHEMA,
    }
    resp = requests.post(f"{FASTCRW_BASE}/scrape", headers=HEADERS, json=payload)
    if resp.status_code != 200:
        return {}
    return resp.json().get("data", {}).get("json", {})

# --- Step 5: Deduplicate by content hash ---

def content_hash(markdown: str) -> str:
    normalized = " ".join(markdown.split())  # collapse whitespace before hashing
    return hashlib.sha256(normalized.encode()).hexdigest()

# --- Full pipeline ---

def run_aggregation(domains: list[str], seen_hashes: set[str]) -> list[dict]:
    results = []

    for domain in domains:
        print(f"Mapping {domain} ...")
        all_urls = map_domain(domain)
        content_urls = [u for u in all_urls if is_content_url(u)]
        print(f"  {len(content_urls)} content URLs (filtered from {len(all_urls)} total)")

        for url in content_urls:
            page = scrape_page(url)
            markdown = page.get("markdown", "")
            if not markdown:
                continue

            h = content_hash(markdown)
            if h in seen_hashes:
                print(f"  [dup] {url}")
                continue

            seen_hashes.add(h)
            metadata = extract_metadata(url)

            results.append({
                "url": url,
                "source_domain": domain,
                "content_hash": h,
                "markdown": markdown,
                "metadata": metadata,
            })
            print(f"  [new] {url} — {metadata.get('headline', '(no headline)')}")

    return results


if __name__ == "__main__":
    domains = [
        "https://techcrunch.com",
        "https://www.theverge.com",
        "https://arstechnica.com",
    ]
    seen: set[str] = set()
    articles = run_aggregation(domains, seen)
    print(f"\nAggregated {len(articles)} unique articles.")
    # → Persist to your database here

3. TypeScript / Node.js integration

This TypeScript snippet shows how to use fastCRW inside a Next.js API route or a Node.js worker to crawl a domain and stream results into a data store:

const FASTCRW_BASE = "https://api.fastcrw.com/v1";
const FASTCRW_KEY = process.env.FASTCRW_API_KEY!;

interface ArticleMetadata {
  headline: string;
  author?: string;
  published_date?: string;
  summary?: string;
  category?: string;
}

interface AggregatedArticle {
  url: string;
  sourceDomain: string;
  contentHash: string;
  markdown: string;
  metadata: ArticleMetadata;
  scrapedAt: string;
}

// Map a domain to get all URLs
async function mapDomain(domain: string): Promise<string[]> {
  const res = await fetch(`${FASTCRW_BASE}/map`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${FASTCRW_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ url: domain }),
  });
  const data = await res.json();
  return data.urls ?? [];
}

// Scrape a URL and extract structured metadata in one call
async function scrapeWithExtraction(url: string): Promise<{
  markdown: string;
  metadata: ArticleMetadata;
} | null> {
  const res = await fetch(`${FASTCRW_BASE}/scrape`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${FASTCRW_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      url,
      formats: ["markdown", "json"],
      jsonSchema: {
        type: "object",
        properties: {
          headline:       { type: "string" },
          author:         { type: "string" },
          published_date: { type: "string" },
          summary:        { type: "string" },
          category:       { type: "string" },
        },
        required: ["headline"],
      },
    }),
  });
  if (!res.ok) return null;
  const body = await res.json();
  return {
    markdown: body.data?.markdown ?? "",
    metadata: body.data?.json ?? {},
  };
}

// Stable SHA-256 hash using Web Crypto (works in Edge and Node 18+)
async function sha256(text: string): Promise<string> {
  const buf = await crypto.subtle.digest(
    "SHA-256",
    new TextEncoder().encode(text.replace(/\s+/g, " "))
  );
  return Array.from(new Uint8Array(buf))
    .map((b) => b.toString(16).padStart(2, "0"))
    .join("");
}

// Full aggregation run for a list of domains
export async function aggregateContent(
  domains: string[],
  seenHashes: Set<string>
): Promise<AggregatedArticle[]> {
  const CONTENT_PATTERNS = ["/blog/", "/article/", "/news/", "/post/", "/story/"];
  const articles: AggregatedArticle[] = [];

  for (const domain of domains) {
    const allUrls = await mapDomain(domain);
    const contentUrls = allUrls.filter((u) =>
      CONTENT_PATTERNS.some((p) => u.includes(p))
    );

    // Scrape up to 10 URLs concurrently to stay within plan concurrency limits
    const chunks = [];
    for (let i = 0; i < contentUrls.length; i += 10) {
      chunks.push(contentUrls.slice(i, i + 10));
    }

    for (const chunk of chunks) {
      const results = await Promise.all(
        chunk.map(async (url) => {
          const scraped = await scrapeWithExtraction(url);
          if (!scraped?.markdown) return null;

          const hash = await sha256(scraped.markdown);
          if (seenHashes.has(hash)) return null; // exact duplicate
          seenHashes.add(hash);

          return {
            url,
            sourceDomain: domain,
            contentHash: hash,
            markdown: scraped.markdown,
            metadata: scraped.metadata,
            scrapedAt: new Date().toISOString(),
          } satisfies AggregatedArticle;
        })
      );

      articles.push(...results.filter((r): r is AggregatedArticle => r !== null));
    }
  }

  return articles;
}

Deduplication Strategies

Content aggregation from multiple sources produces duplicates at every level. A robust pipeline handles three distinct cases:

Exact duplicates (same bytes)

The simplest case: syndicated content copied verbatim across sites. Two articles with the same body text will hash identically once normalized (whitespace collapsed, HTML stripped). Use SHA-256 of the normalized markdown body. fastCRW's consistent markdown output makes this more reliable than hashing raw HTML, because HTML structure varies per site even when body text is identical.

Implementation: after scraping, normalize the markdown (strip frontmatter, collapse runs of whitespace, lowercase), compute SHA-256, store in a content_hashes table, and skip any URL whose hash already exists.

Near-duplicates (paraphrased or lightly edited)

Syndication services often lightly rewrite content. Two articles covering the same story with 80% word overlap will have different SHA-256 hashes but should still be treated as duplicates in a curated feed.

Implementation: embed each article body using a text embedding model (OpenAI text-embedding-3-small, open-source bge-m3, or similar). Store embeddings in a vector database (pgvector, Qdrant, Chroma). At ingest time, query for the nearest neighbor — if cosine similarity exceeds 0.92 (a common threshold for near-dup detection), flag the new article as a near-duplicate rather than inserting it.

URL canonicalization

The same article often appears at multiple URLs (/article/slug, /article/slug/, ?utm_source=twitter, https://www.site.com vs https://site.com). Resolve these before scraping to avoid redundant credit spend.

Implementation:

1. Strip all query parameters except pagination params
2. Normalize trailing slashes
3. Resolve www to root domain (or vice versa, consistently)
4. Follow HTTP 301/302 redirects before inserting a URL into your scrape queue
5. Store the canonical URL (after redirect resolution) as the primary key

Source attribution and republishing ethics

Aggregation pipelines that surface content to end users carry attribution obligations independent of legal requirements:

• Always persist the source URL and domain alongside every scraped document
• Display source credit and a link to the original in any downstream UI
• For verbatim text, link to the original rather than republishing full body text unless you have a license to do so
• robots.txt is fastCRW's default starting point — it can be overridden only when the operator has the legal right to access the content
• Honor DMCA takedown requests promptly; build a removal flow into your pipeline from day one

fastCRW vs. Firecrawl vs. Apify for Content Aggregation

Criterion	fastCRW	Firecrawl	Apify
API compatibility	Firecrawl-compatible — same endpoint names, same request shape	Firecrawl (reference)	Proprietary Actor SDK
Self-hosting	Single ~8 MB static binary, 1 container (AGPL-3.0, CANONICAL-FACTS.md §7, structural fact)	~2–3 GB Docker pull, 5 containers	Apify platform only (no self-host)
Cost on your infra	$0 per 1,000 scrapes — you pay only your server (CANONICAL-FACTS.md §8)	$0.83–5.33 per 1,000 scrapes on managed cloud (competitor-prices.lock.md, verified 2026-05-18)	Consumption-based on Apify platform; no self-host option
Output format	Clean markdown + JSON extraction	Clean markdown + JSON extraction	Varies by Actor; no standard output format
MCP server built-in	Yes — POST /mcp streamable HTTP (CANONICAL-FACTS.md §4)	No built-in MCP server	No built-in MCP server
JS rendering	Chrome + LightPanda, auto-fallback	Chrome-based	Full browser via Playwright/Puppeteer Actors
LLM extraction	JSON schema via /v1/scrape with formats: ["json"]; 5 credits (CANONICAL-FACTS.md §3)	JSON schema via /v1/scrape and /v1/extract	Depends on the Actor
Search endpoint	/v1/search with optional LLM answer synthesis (CANONICAL-FACTS.md §6)	/v1/search	Via Google SERP Actors
Accuracy (819 labeled URLs)	63.74% truth-recall — highest of three tested (diagnose_3way.py, 2026-05-08, CANONICAL-FACTS.md §5)	56.04% truth-recall (same benchmark)	Not tested in benchmark
p50 scrape latency	1,914 ms (CANONICAL-FACTS.md §5, 2026-05-08)	2,305 ms	Not benchmarked
Drop-in migration	Change base URL only	Reference	Requires Actor rewrite

When to pick fastCRW over Firecrawl: you want to self-host for cost or data-residency reasons, need the built-in MCP server, or are migrating from Firecrawl and want a drop-in swap. See the Firecrawl alternatives page for the full migration guide.

When to consider Apify: you need a large ecosystem of pre-built site-specific scrapers (Amazon, LinkedIn, etc.) and are comfortable with proprietary platform lock-in. Apify's Actor marketplace covers specialized sources that general-purpose scrapers handle poorly. For generic content aggregation from public web pages, fastCRW is simpler and cheaper to operate.

Production Considerations

Scaling throughput

Concurrency limits determine your maximum throughput. Each paid plan allows a fixed number of concurrent requests (see /pricing for current limits). For bulk aggregation jobs:

• Batch URLs into chunks matching your plan's concurrency limit
• Use a job queue (BullMQ, Celery, Sidekiq) to manage parallel requests without overrunning limits
• POST /v1/crawl handles internal concurrency for you — prefer it over many parallel /v1/scrape calls when you are crawling a single domain
• For cross-domain aggregation at high volume, issue concurrent /v1/scrape calls — one per domain at a time — to respect per-domain rate limits

Handling site changes and rendering modes

Not all aggregation sources are equally cooperative:

• Plain static HTML sites (renderer: "http" or default auto): 1 credit per page. Most blogs, news archives, and documentation sites work here.
• JavaScript-rendered content (renderer: "lightpanda" or "chrome"): needed for React/Vue/Next.js sites, dynamic loading, or infinite scroll. Both LightPanda and Chrome cost 1 credit per page; Chrome handles the most complex rendering.
• Paywalled or login-gated content: fastCRW does not support session authentication — if your target sources require login, you need a credentials-passing layer or a purpose-built private scraper.
• Anti-bot protections: fastCRW uses a chrome-stealth fallback path. For aggressive bot detection, respect rate limits and space requests appropriately. The same fallback that handles complex rendering produces fastCRW's higher p90 latency (14,157 ms — disclosed honestly from the diagnose_3way.py benchmark, 2026-05-08).

Storage and indexing

A content aggregation database needs at minimum:

CREATE TABLE articles (
  id              BIGSERIAL PRIMARY KEY,
  url             TEXT UNIQUE NOT NULL,
  source_domain   TEXT NOT NULL,
  content_hash    CHAR(64) NOT NULL,      -- SHA-256 hex
  markdown        TEXT NOT NULL,
  headline        TEXT,
  author          TEXT,
  published_date  TIMESTAMPTZ,
  summary         TEXT,
  category        TEXT,
  scraped_at      TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at      TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX idx_articles_content_hash ON articles (content_hash);
CREATE INDEX idx_articles_source_domain ON articles (source_domain);
CREATE INDEX idx_articles_scraped_at ON articles (scraped_at DESC);
-- For full-text search:
ALTER TABLE articles ADD COLUMN fts tsvector
  GENERATED ALWAYS AS (to_tsvector('english', coalesce(headline,'') || ' ' || coalesce(summary,'') || ' ' || coalesce(markdown,''))) STORED;
CREATE INDEX idx_articles_fts ON articles USING GIN (fts);

For vector search (near-dup detection + semantic retrieval), add a pgvector column alongside the SHA-256 hash column.

Re-crawl strategy

Not all sources update at the same rate. A tiered re-crawl schedule balances freshness against credit spend:

Source tier	Re-map frequency	Re-scrape frequency
Breaking news / live blogs	Weekly	Every 1–4 hours
Daily news sources	Weekly	Daily
Blog / long-form content	Monthly	Weekly (on new URLs only)
Documentation / evergreen	Quarterly	Monthly (on changed URLs)
Reference data / static archives	Once	On-demand only

"Changed URLs only" means: compare the new content hash against the stored hash — only process articles where the hash differs. This makes re-crawl cheap: you spend 1 credit per page to fetch and compare, then 5 credits only on pages that actually changed and need fresh metadata extraction.

Good Fits

• Multi-source content products: aggregating tech blogs, trade publications, or niche forums into a unified reader
• Data pipeline dashboards: feeding scraped content into BI tools, Metabase, or custom dashboards for editorial or research teams
• RAG knowledge bases: building a domain-specific corpus for retrieval-augmented generation (RAG pipelines)
• LLM training data: curating high-quality web content at scale for fine-tuning datasets
• Market research platforms: pulling product announcements, press releases, and analyst reports across sectors
• AI agent pipelines: equipping AI agents with live web context via the MCP integration
• News aggregation: the specific sub-case of aggregating news sites — see news aggregation for a news-focused deep dive

When to Pick Something Else

Web scraping is the right tool when content is only available as a rendered web page. In these cases, use the native alternative instead:

• The source has a well-maintained API: use the API directly (REST, GraphQL, SDK). It will be faster, more reliable, and ethically cleaner.
• The source has a data export: CSV/JSON data dumps from public datasets (government data, academic repos) are more reliable than scraping.
• You need real-time streaming: scraping is batch-oriented. For event streams or live feeds (sports scores, financial ticks), use native websocket or streaming APIs.
• The source is aggressively gated: if a site blocks automated access entirely and has no public API, the legal and technical cost of circumventing protections usually outweighs the benefit. Focus your aggregation on sources with permissive robots.txt and terms of service.

Related Resources

• News aggregation — news-specific pipeline with breaking-news detection and per-source freshness tuning
• RAG pipelines — pipe aggregated content into a vector store for retrieval-augmented chat
• Firecrawl alternatives — full migration guide if you are coming from Firecrawl
• Apify alternatives — comparison when you need the Actor ecosystem vs. a general-purpose scraping API
• MCP integration — connect fastCRW directly to Claude Code, Cursor, or any MCP client for agent-driven content access
• Pricing — live plan tiers and credit costs