Web Scraping for Market Research

Why Market Research Needs Web Scraping

Manual competitor monitoring has a ceiling. An analyst can visit 10 competitor sites per week, update a spreadsheet, and write a monthly report. But competitive markets don't move on analyst schedules — a competitor can reprice, ship a feature, or launch a new product tier between Monday's check and Friday's report.

Automated scraping removes the ceiling. You define the sources and signals once; the pipeline runs every day. The analyst's job shifts from data collection to interpretation — which is where their judgment actually adds value.

What automated market research gives you:

• Continuous coverage across dozens of competitor sites, review platforms, job boards, and industry directories — without analyst time per source
• Structured data that flows into dashboards and data warehouses instead of spreadsheets
• Historical baselines for trend detection — not just "their pricing changed" but "their entry tier increased 31% over 6 months"
• Scale to monitor 100+ sources simultaneously, covering far more of the competitive surface than any manual process

What Market Research Teams Monitor

Signal type	Source pages	Extraction goal
Competitor pricing	/pricing, /plans, /packages pages	Plan names, price points, feature inclusions per tier
Feature releases	Changelog, release notes, blog	New feature names, release dates, positioning language
Product messaging	Homepage, hero copy, taglines	Value proposition changes, target audience shifts
Customer sentiment	G2, Capterra, Trustpilot, Reddit	Rating scores, review themes, feature requests
Hiring signals	Careers pages, Greenhouse/Lever	Open roles by department — infers investment priorities
Content strategy	Blog, docs, resource center	Topic focus, SEO targeting, thought leadership positioning
Partnership ecosystem	Integrations pages, partner directories	Which tools they're betting on, which they're dropping

Where fastCRW Fits

Research need	fastCRW endpoint	Notes
Discover all pages on a competitor domain	/v1/map	Full URL inventory — find pricing, blog, changelog, careers
Scrape and extract structured data	/v1/scrape + jsonSchema	Pull price fields, feature names, ratings into clean JSON
Crawl an entire competitor blog section	/v1/crawl	Set maxDepth + maxPages to cap scope per competitor
Search for competitor mentions across the web	/v1/search	Find press coverage, forums, review sites mentioning a brand
Render JS-heavy SPAs and dynamic pricing pages	Auto renderer	Chrome fallback for fully dynamic pages (1 credit, same as any renderer)

Typical Market Research Pipeline

A production competitive intelligence pipeline has four layers:

1. Discovery layer Use /v1/map on each competitor domain monthly to discover their full URL structure. Filter for high-signal page types: pricing, changelog, product feature pages, and careers. Cache URL lists for 30 days — competitor site structures are stable.

2. Scheduled scraping layer Schedule /v1/scrape on each target URL at appropriate frequency (daily for pricing, weekly for features, twice-weekly for blog). Store raw markdown alongside extracted JSON — qualitative analysis of phrasing changes requires the full text.

3. Change detection layer On each new scrape, compare content hash against the stored version. When hashes differ, run a field-level diff on extracted JSON. Meaningful changes (price field, tier name, feature list) trigger alerts to Slack or email. Small changes (a word in a paragraph) are logged but not alerted.

4. Aggregation and reporting layer Push extracted fields to a data warehouse (BigQuery, Snowflake, Postgres). Build BI dashboards showing pricing trends over time, feature gap matrices, and sentiment averages by quarter.

Implementation: Competitive Intelligence Pipeline

curl — discover all pages on a competitor domain

curl -X POST https://api.fastcrw.com/v1/map \
  -H "Authorization: Bearer $CRW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://competitor.com"}'

curl — extract pricing data with a JSON schema

curl -X POST https://api.fastcrw.com/v1/scrape \
  -H "Authorization: Bearer $CRW_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://competitor.com/pricing",
    "formats": ["json"],
    "jsonSchema": {
      "type": "object",
      "properties": {
        "plans": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "name":     { "type": "string" },
              "price":    { "type": "string" },
              "billing":  { "type": "string", "description": "monthly or annual" },
              "features": { "type": "array", "items": { "type": "string" } }
            }
          }
        },
        "currency":        { "type": "string" },
        "last_updated":    { "type": "string" }
      }
    }
  }'

Python — full competitor monitoring pipeline

import requests
import hashlib
import json
from datetime import datetime
from typing import Optional

CRW_API_KEY = "your-api-key"
CRW_BASE_URL = "https://api.fastcrw.com/v1"

PRICING_SCHEMA = {
    "type": "object",
    "properties": {
        "plans": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "name":     { "type": "string" },
                    "price":    { "type": "string" },
                    "billing":  { "type": "string" },
                    "features": { "type": "array", "items": { "type": "string" } }
                }
            }
        },
        "currency": { "type": "string" }
    }
}

SENTIMENT_SCHEMA = {
    "type": "object",
    "properties": {
        "reviews": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "rating":        { "type": "number" },
                    "summary":       { "type": "string" },
                    "pros":          { "type": "string" },
                    "cons":          { "type": "string" },
                    "reviewer_role": { "type": "string" },
                    "date":          { "type": "string" }
                }
            }
        },
        "overall_rating": { "type": "number" },
        "total_reviews":  { "type": "number" }
    }
}

def map_domain(domain: str) -> list[str]:
    resp = requests.post(
        f"{CRW_BASE_URL}/map",
        json={"url": f"https://{domain}"},
        headers={"Authorization": f"Bearer {CRW_API_KEY}"},
        timeout=30
    )
    resp.raise_for_status()
    return resp.json().get("urls", [])

def scrape_page(url: str, schema: Optional[dict] = None) -> dict:
    payload: dict = {"url": url}
    if schema:
        payload["formats"] = ["json", "markdown"]
        payload["jsonSchema"] = schema
    else:
        payload["formats"] = ["markdown"]

    resp = requests.post(
        f"{CRW_BASE_URL}/scrape",
        json=payload,
        headers={"Authorization": f"Bearer {CRW_API_KEY}"},
        timeout=30
    )
    resp.raise_for_status()
    return resp.json().get("data", {})

def content_hash(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()

def detect_pricing_page(urls: list[str]) -> Optional[str]:
    """Find a pricing page URL in a discovered URL list."""
    for url in urls:
        path = url.lower()
        if any(kw in path for kw in ["/pricing", "/plans", "/packages"]):
            return url
    return None

def monitor_competitor(domain: str, stored_hashes: dict[str, str]) -> dict:
    """Run competitive monitoring for one domain. Returns changed signals."""
    changes: dict = {"domain": domain, "checked_at": datetime.utcnow().isoformat(), "changes": []}

    # Discover pages
    urls = map_domain(domain)
    pricing_url = detect_pricing_page(urls)

    if pricing_url:
        # Scrape pricing with extraction
        data = scrape_page(pricing_url, PRICING_SCHEMA)
        markdown = data.get("markdown", "")
        new_hash = content_hash(markdown)
        old_hash = stored_hashes.get(pricing_url)

        if old_hash is None:
            changes["changes"].append({
                "type": "first_scrape",
                "url": pricing_url,
                "data": data.get("json")
            })
        elif new_hash != old_hash:
            changes["changes"].append({
                "type": "pricing_change",
                "url": pricing_url,
                "data": data.get("json"),
                "previous_hash": old_hash
            })

        stored_hashes[pricing_url] = new_hash

    return changes

def run_market_intelligence(competitors: list[str], stored_hashes: dict) -> list[dict]:
    """Monitor a list of competitor domains."""
    all_changes = []
    for domain in competitors:
        print(f"Monitoring {domain}...")
        try:
            result = monitor_competitor(domain, stored_hashes)
            if result["changes"]:
                print(f"  {len(result['changes'])} change(s) detected!")
                all_changes.append(result)
        except Exception as e:
            print(f"  Error: {e}")
    return all_changes

if __name__ == "__main__":
    competitors = [
        "firecrawl.dev",
        "apify.com",
        "scrapfly.io",
        "brightdata.com",
    ]

    # In production, load/save stored_hashes from your database
    stored_hashes: dict = {}
    changes = run_market_intelligence(competitors, stored_hashes)

    print(f"\n=== INTELLIGENCE REPORT ({datetime.utcnow().date()}) ===")
    if not changes:
        print("No changes detected since last run.")
    for change in changes:
        print(f"\n{change['domain']}:")
        for c in change["changes"]:
            print(f"  [{c['type'].upper()}] {c['url']}")

JavaScript — search-driven market research

const CRW_API_KEY = process.env.CRW_API_KEY!;
const CRW_BASE_URL = "https://api.fastcrw.com/v1";

// Use /v1/search to find competitor mentions, press coverage, and reviews
async function searchCompetitorMentions(competitorName: string): Promise<unknown[]> {
  const queries = [
    `${competitorName} pricing 2026`,
    `${competitorName} review site:g2.com OR site:capterra.com`,
    `${competitorName} vs alternatives`,
    `"${competitorName}" new features`,
  ];

  const results: unknown[] = [];
  for (const q of queries) {
    const res = await fetch(`${CRW_BASE_URL}/search`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${CRW_API_KEY}`,
        "Content-Type": "application/json",
      },
      body: JSON.stringify({ query: q, limit: 5 }),
    });
    const data = await res.json();
    results.push(...(data.results ?? []));
  }
  return results;
}

// Then scrape the top results for full content
async function scrapeSearchResults(urls: string[]): Promise<unknown[]> {
  const scrapes = await Promise.allSettled(
    urls.map((url) =>
      fetch(`${CRW_BASE_URL}/scrape`, {
        method: "POST",
        headers: {
          Authorization: `Bearer ${CRW_API_KEY}`,
          "Content-Type": "application/json",
        },
        body: JSON.stringify({ url, formats: ["markdown"] }),
      }).then((r) => r.json())
    )
  );
  return scrapes
    .filter((r) => r.status === "fulfilled")
    .map((r) => (r as PromiseFulfilledResult<unknown>).value);
}

// Example usage
(async () => {
  const mentions = await searchCompetitorMentions("Firecrawl");
  const urls = (mentions as { url: string }[]).slice(0, 10).map((r) => r.url);
  const content = await scrapeSearchResults(urls);
  console.log(`Collected ${content.length} pages of competitor coverage`);
})();

Use Case Deep Dives

Competitor pricing intelligence

Pricing pages are the highest-value target for market research teams. A competitor's pricing page encodes their go-to-market strategy: who they're targeting (price points), how they think about value (what's in each tier), and how they're growing (new enterprise tiers, usage-based models).

Set up daily scrapes of all competitor pricing pages. Extract plan names, price points, billing cycle (monthly/annual), and the feature list per tier with a jsonSchema. Store results in a time-series table. When a hash change is detected, run a field-level diff and send the change to your Slack channel with before/after values.

Over 6 months of daily scraping, you'll have a complete pricing history for every competitor — invaluable context for your own repricing decisions.

Sentiment analysis from review platforms

G2, Capterra, and Trustpilot are goldmines for competitive intelligence. Each review encodes a customer's real experience — what they loved, what failed, and what they switched from.

Scrape competitor product pages on G2 and Capterra monthly. Use LLM extraction to pull rating, review text, pros/cons, and reviewer role (VP vs. individual contributor). Aggregate across hundreds of reviews to surface recurring themes: which feature gets complained about most, which use case competitors are winning. That's your product gap analysis.

Hiring signal tracking

A competitor's job board predicts their roadmap 6–12 months in advance. Five new ML engineer postings means an AI feature is coming. Three open positions for "Enterprise Sales Engineer" means they're going upmarket. A wave of customer success hires means their churn is rising.

Crawl competitor careers pages weekly with /v1/crawl. Extract job title, department, and location. Track the delta week over week — new postings signal expansion; closed postings signal a pivot or hiring freeze.

Content strategy and SEO mapping

Crawl a competitor's blog with /v1/crawl (set maxDepth: 2, maxPages: 500). Extract article titles, publish dates, and topic keywords. This gives you a 1-2 year map of what they're betting on in content — which keywords they're targeting, which use cases they're pushing, where they're investing in thought leadership. Map that against your own content plan to find gaps and opportunities.

Production Considerations

Change alerting without alert fatigue

Not every diff is worth an alert. Implement a significance filter: only alert when a price field changes, a tier name changes, or a feature list adds/removes items. Ignore CSS changes, date/time updates in footers, and A/B test variations. A tiered alert system (high-priority Slack channel for pricing, daily digest for content changes) reduces noise.

Handling JavaScript-heavy pricing pages

Modern SaaS pricing pages are increasingly React-driven SPAs that render price values client-side. fastCRW's Chrome renderer handles these fully (1 credit — every renderer costs the same). When you see empty or placeholder pricing data in markdown, switch to renderer: "chrome" for that URL.

Storing historical baselines

The value of market research data compounds over time. Store every scrape in a time-series table with url, scraped_at, content_hash, markdown_text, and extracted_json columns. This gives you a queryable history to answer questions like "when did Competitor X add an enterprise tier?" or "how has their hero copy evolved this year?"

Respecting ToS and robots.txt

fastCRW respects robots.txt by default. For market research, you're almost always scraping publicly marketed pages (pricing, blog, product) that companies want the world to see. Still, verify each source's ToS before automating at scale. Sites behind authenticated sessions or enterprise portals require different approaches and likely explicit permission.

Data freshness and scrape scheduling

Don't scrape everything at the same frequency. Signal-based scheduling saves credits and reduces load on target sites:

Signal type	Recommended frequency	Rationale
Pricing pages	Daily	Can change overnight; high business impact
Product feature pages	Twice weekly	Release cadences are predictable
Changelog / release notes	Twice weekly	Tied to release schedules
Blog and content	Weekly	Content strategy moves slowly
Careers pages	Daily	Hiring signals are time-sensitive
Review platforms	Monthly	Aggregate scores change slowly

Credit Cost Estimates

All credit costs from CANONICAL-FACTS §3 (marketing/CANONICAL-FACTS.md, verified 2026-05-29):

Small team competitive monitoring (5 competitors, 10 pages each):

• Daily pricing scrapes: 5 domains × 1 page × 1 credit × 30 days = 150 credits/month
• Weekly feature page scrapes: 5 domains × 5 pages × 1 credit × 4 weeks = 100 credits/month
• Monthly map scans: 5 domains × 1 credit = 5 credits/month
• Pricing extraction (JSON): 5 domains × 4 scrapes/month × 5 credits = 100 credits/month
• Total: ~355 credits/month → well within the Free tier (500 one-time lifetime credits) for initial setup; Hobby plan ($13/mo, 3,000 credits) for ongoing

Mid-size market research (20 competitors, 50 pages each, daily pricing + weekly feature):

• Daily pricing: 20 × 5 × 30 = 3,000 credits/month
• Weekly features: 20 × 25 × 4 = 2,000 credits/month
• Extraction (JSON): 20 × 30 × 5 = 3,000 credits/month
• Map scans: 20 × 1 = 20 credits/month
• Total: ~8,020 credits/month → Standard plan ($69/mo — launch price, was $99, 100,000 credits)

Enterprise market intelligence (100 sources, real-time change detection):

• 100 sources × 10 pages × daily scrapes × 30 = 30,000 credits/month (markdown)
• Extraction on changed pages only (10% change rate): 30,000 × 0.1 × 5 = 15,000 credits
• Sentiment scraping (50 review pages monthly): 50 × 5 = 250 credits
• Total: ~45,000–60,000 credits/month → Standard plan ($69/mo, 100,000 credits)

Pricing derives from PLAN_DISPLAY in src/lib/plans-client.ts. Check /pricing for current rates.

For very-large enterprise workloads, self-hosting eliminates per-scrape credit costs entirely — you pay only your server.

Good Fits

• Product teams tracking competitor feature launches to inform roadmap prioritization
• Pricing and revenue teams monitoring market rates to calibrate their own tier structure
• Strategy and GTM teams mapping the full competitive landscape across 20+ players
• Content and SEO teams analyzing competitor content strategy and keyword targeting
• Customer success teams monitoring review platforms for competitor weaknesses to counter in sales conversations
• Investment and VC teams monitoring portfolio company competitors at scale
• Research and analyst teams building quantitative datasets on market trends

When to Pick Something Else

• Data behind login walls or authenticated APIs: fastCRW cannot access content that requires a session without credentials. Use the platform's official API or a data partnership.
• Twitter/X and LinkedIn data: Both platforms prohibit scraping in their ToS. Use their official APIs or a licensed data provider.
• Pre-built competitive intelligence dashboards: If you want a turnkey solution with built-in alerting, CRM integration, and battlecard generation, a dedicated platform (Crayon, Klue) is faster to deploy — at significantly higher cost.
• TV, radio, and print media monitoring: Traditional media requires specialist providers. fastCRW covers the public web only.

Related Resources

• Competitor monitoring — the focused version: real-time change tracking on a defined competitor set
• Brand monitoring — track mentions of your own brand across the same public web sources
• Price monitoring — dedicated pricing intelligence pipeline with change-detection patterns
• Firecrawl alternative — accuracy and cost comparison for research-scale workloads
• Apify alternative — full actor platform vs. simple scraping API for research use cases
• MCP integration — use fastCRW tools directly from Claude or AI agent frameworks
• Benchmarks — accuracy and latency data so you can evaluate fastCRW against alternatives
• Pricing — current plan credits and rates