Build a Competitor Monitoring Tool (Dashboard)

By the fastCRW team · Credit costs and footprint verified against the canonical fact sheet 2026-05-18 · See /pricing for current rates · Verify independently before relying on any number.

Disclosure: we build fastCRW. This is a vendor-authored tutorial, so weight it accordingly — the architecture below works on any Firecrawl-compatible API, and we call out where Firecrawl genuinely wins.

Build a competitor monitoring tool that pays for itself

A competitor monitoring tool watches a handful of rival pages — pricing, features, changelog, blog — and tells you the moment something changes. The hard part is not the first scrape; it is the thousandth. You re-scrape the same pages forever, so per-page cost and infrastructure footprint dominate the economics, and a flaky diff that flaps on cosmetic HTML noise will train your team to ignore the alerts. This tutorial builds the whole loop in Python: collect pages, diff them, summarize what changed with an LLM, and surface it in a dashboard — with honest cost math at every step.

The design rides on one structural fact: fastCRW meters a scrape or crawl page at a flat 1 credit regardless of renderer — http, lightpanda, or chrome — with no ScrapingBee-style 5× JavaScript multiplier (fastCRW canonical credit table, verified 2026-05-18). Continuous monitoring is exactly the workload where flat per-page metering and a small footprint matter most.

What a competitor monitoring tool watches

Scope the tool before you write code. Most teams watch four page types, and each has a different change cadence:

• Pricing pages — low change frequency, high business impact. A tier price moving is a board-meeting event.
• Feature / product pages — medium frequency; new capabilities and positioning shifts.
• Changelog / release notes — high frequency, append-only; the richest signal for "what are they shipping?"
• Blog / announcements — high frequency, noisy; useful for narrative, bad for alerting unless filtered.

Page-change detection vs full re-scrape

You have two strategies. Full re-scrape pulls every watched URL on each run and diffs the result — simple, deterministic, and the right default for a watchlist of dozens of pages. Change detection tries to fetch only what moved (cheap HEAD checks, sitemap lastmod, ETags) before scraping. Start with full re-scrape; it is one mental model and the per-page cost is flat, so the savings from cleverness rarely justify the complexity until your watchlist is in the hundreds.

Step 1: Collect competitor pages

First discover the pages worth watching, then scrape them to a stable text format. Use /v1/map (1 credit) to enumerate a competitor's URLs, pick the ones that matter, then scrape each to markdown. Clean markdown — not raw HTML — is what makes the diff stable later.

from crw import CrwClient client = CrwClient(api_url="https://api.fastcrw.com", api_key="YOUR_KEY") # Discover URLs once, then hand-pick the watchlist (1 credit per /v1/map call). site_map = client.map(url="https://competitor.example") watchlist = [u for u in site_map["links"] if any(k in u for k in ("/pricing", "/changelog", "/features"))] def snapshot(url: str) -> str: # 1 credit on any renderer (http, lightpanda, or chrome — all flat 1). res = client.scrape(url=url, formats=["markdown"]) return res["markdown"]

Because fastCRW is Firecrawl-compatible, the same code runs against Firecrawl by changing api_url — the watchlist is portable. The auto renderer picks chrome → lightpanda → http, so JS-heavy pricing pages still render, and every page costs the same flat 1 credit regardless of which renderer fires.

Step 2: Detect and store changes

fastCRW is stateless per request — it does not remember yesterday's scrape (fastCRW canonical honest-gaps list). That is by design, and it means you own the history store. This is the single most important architectural fact in the whole tool: the engine fetches, you persist. A tiny SQLite table is plenty to start.

import sqlite3, hashlib, datetime, difflib db = sqlite3.connect("monitor.db") db.execute("""CREATE TABLE IF NOT EXISTS snapshots( url TEXT, captured_at TEXT, content_hash TEXT, body TEXT)""") def record(url: str, body: str): h = hashlib.sha256(body.encode()).hexdigest() prev = db.execute( "SELECT content_hash, body FROM snapshots WHERE url=? " "ORDER BY captured_at DESC LIMIT 1", (url,)).fetchone() if prev and prev[0] == h: return None # unchanged — no alert, no LLM call db.execute("INSERT INTO snapshots VALUES (?,?,?,?)", (url, datetime.datetime.utcnow().isoformat(), h, body)) db.commit() if not prev: return None # first capture is a baseline, not a "change" diff = "\n".join(difflib.unified_diff( prev[1].splitlines(), body.splitlines(), lineterm="")) return diff

Hashing the markdown gives you a free no-op fast path: if the hash matches the last snapshot, you skip the diff and — crucially — skip the LLM call in Step 3, so unchanged pages cost nothing beyond the scrape. Diffing markdown rather than HTML is what keeps the signal clean: a re-ordered <div> or a changed analytics tag does not move the rendered text, so your alerts fire on substance, not cosmetic churn.

Step 3: Summarize changes with an LLM

A raw unified diff is noise to a human reader. Hand the diff to an LLM and ask it to answer two questions: what changed and why it matters. On fastCRW Cloud, managed answer mode (paid plans, no key of your own) runs the request on fastCRW's managed LLM, metered in credits based on usage and capped per request. LLM features require a paid plan. The same managed LLM also powers fastCRW's first-party LLM extraction on paid plans.

SUMMARY_PROMPT = """You are a competitive-intelligence analyst. Given this unified diff of a competitor's {page_type} page, output: 1. What changed (one line). 2. Why it matters to us (one line, or "low signal"). Diff: {diff}""" def summarize(diff: str, page_type: str, llm) -> str: return llm.complete(SUMMARY_PROMPT.format(page_type=page_type, diff=diff))

Because Step 2 only calls this on real changes, the LLM bill scales with how often competitors actually move — not with how often you poll.

Step 4: Build the dashboard and alerts

The dashboard is a thin read layer over your snapshot store. A single-file Streamlit app is the fastest path: list watched URLs, show the most recent change summary per URL, and let you click through to the full diff.

import streamlit as st st.title("Competitor Monitor") rows = db.execute( "SELECT url, captured_at, body FROM snapshots ORDER BY captured_at DESC").fetchall() for url, ts, _ in rows[:50]: st.write(f"**{url}** — last change {ts}")

For alerting, post the LLM summary to a Slack incoming webhook (or any webhook) the moment record() returns a non-null diff. To avoid alert fatigue, gate notifications on the LLM's "why it matters" line — suppress anything it scores as "low signal," and batch blog-page changes into a daily digest while sending pricing-page changes immediately.

import requests def alert(url: str, summary: str): if "low signal" in summary.lower(): return # batch into the daily digest instead requests.post(SLACK_WEBHOOK_URL, json={"text": f":eyes: {url}\n{summary}"})

Run it on a schedule

The whole loop is a stateless script, so cron (or GitHub Actions, or a systemd timer) is all the scheduling you need. Each run maps nothing (you cache the watchlist), scrapes each watched URL, diffs, and conditionally summarizes. See scheduled crawls with cron for the scheduling patterns in depth.

Cost and self-host vs managed

Here is the honest math for a watchlist of 50 pages polled twice a day, assuming the pages render on http/lightpanda (1 credit each):

That ~3,000 scrape-credits/month sits inside the Hobby tier — every page costs 1 credit flat no matter which renderer is used, so JS-only pricing pages carry no surcharge. See live numbers on /pricing rather than trusting a table that can drift.

The bigger lever is self-hosting. The fastCRW engine is a single ~8 MB Rust binary in one container (a structural fact, not a benchmark), licensed AGPL-3.0, so you can self-host it for $0 in software cost and pay only for the server (fastCRW canonical footprint + license facts). For a monitoring tool that re-scrapes the same pages indefinitely, self-hosting turns a recurring per-page bill into a fixed VPS line item, and it keeps your competitor watchlist private — your list of who you watch never leaves your infrastructure. For the managed path with no ops, point the same code at api.fastcrw.com; the only change is the base URL.

Why build your own dashboard instead of using the built-in monitor

fastCRW also ships a built-in /v1/monitor endpoint for scheduled change-detection and webhooks, plus a 12-signal block-detection classifier, user-agent rotation, and stealth fingerprints for anti-bot, wired into the auto escalation ladder. This tutorial builds your own history store and diff logic instead, because a custom dashboard gives you full control over storage, alert routing, and how diffs get presented to your team — fastCRW is stateless per request by design, so you own that layer either way.

What you get in return is a tool whose ongoing cost has a floor, whose diffs are clean because they run on markdown, and whose watchlist stays on your own infrastructure when self-hosted.

Sources

• fastCRW canonical fact sheet — credit costs, structural footprint, managed answer mode, AGPL-3.0/license, verified 2026-05-18.
• fastCRW repo and live pricing: github.com/us/crw · /pricing
• ScrapingBee JavaScript-rendering multiplier (5× JS, 25–75× premium/stealth tiers): scrapingbee.com/pricing

Related: Competitor monitoring with CRW · Scheduled crawls with cron · Build an AI price tracker · Data enrichment guide