I'm building a pipeline for recurring crawls and change detection across thousands of URLs, any recommendations?

Yes — for thousands of URLs, the key is to treat crawling as a data pipeline, not just a fetch loop. A good setup usually has these pieces:

1) Split the system into stages

Use a pipeline like:

1. URL discovery / scheduling
2. Fetch
3. Normalize / clean
4. Detect change
5. Store history
6. Alert / downstream processing

This separation makes retries, scaling, and debugging much easier.

---

2) Use a priority scheduler, not a fixed cron

For recurring crawls, different URLs should have different revisit rates.

Good scheduling inputs:

• historical change frequency
• page importance / business value
• last-modified hints
• error rate / instability
• crawl cost

A simple strategy:

• crawl fast-changing pages more often
• crawl stable pages less often
• back off on repeated failures
• randomize exact timing to avoid burst load

If you’re scaling up, store crawl state in a queue/table with fields like:

• url
• next_crawl_at
• priority
• etag
• last_modified
• last_hash
• fail_count
• last_status

---

3) Use conditional requests whenever possible

For HTTP pages, always try:

• If-None-Match with ETag
• If-Modified-Since with Last-Modified

This saves bandwidth and reduces load. Even if a server supports only one of them, it’s still useful.

---

4) Normalize content before comparing

Raw HTML changes constantly for irrelevant reasons.

Before diffing, normalize:

• remove scripts/styles
• strip tracking params from URLs
• collapse whitespace
• sort attributes if needed
• remove timestamps / dynamic tokens if they’re noisy
• canonicalize text
• ignore ad blocks/nav/footer if possible

For many use cases, store multiple representations:

• raw response
• clean text
• DOM-structured snapshot
• extracted entities/fields

Then compare at the right layer.

---

5) Prefer semantic change detection over byte diff

A byte-level diff produces too many false positives.

Better options:

• hash of normalized text
• block-level DOM hashing
• template-aware extraction
• field-level comparison for structured pages
• simhash/minhash for near-duplicate detection

If pages have a known structure, field-based diffing is usually best:

• title changed?
• price changed?
• availability changed?
• publication date changed?

---

6) Separate “page changed” from “meaningful change”

Not every change matters.

You may want to classify:

• HTML noise only
• cosmetic content change
• meaningful text change
• critical field change
• page removed / redirect / canonicalized

This reduces alert fatigue dramatically.

---

7) Use a storage model that supports versioning

For change detection, keep history.

Recommended storage pattern:

• a current-state table
• a versions/history table
• optional blob/object store for raw snapshots

Example version record:

• url
• fetched_at
• http_status
• content_hash
• normalized_hash
• content_type
• etag
• last_modified
• snapshot_location
• change_type
• diff_summary

---

8) Build idempotent workers

At scale, retries happen. Workers should be safe to rerun.

Make sure:

• duplicate fetches don’t corrupt state
• writes are versioned or atomic
• jobs can be retried after crashes
• deduplication is handled by URL+timestamp or job ID

---

9) Handle rate limits and crawl politeness

For external sites, respect:

• robots.txt if applicable to your policy
• per-domain concurrency limits
• request delays
• retry-after headers
• backoff on 429/503

Use domain-level throttles:

• max concurrent requests per host
• max requests per minute per host
• exponential backoff on errors

This also improves stability.

---

10) Make crawling distributed, but keep coordination centralized

A common approach:

• centralized scheduler / queue
• distributed fetch workers
• shared state store (DB/Redis)
• object storage for snapshots

Good queue options:

• Kafka
• SQS
• RabbitMQ
• Redis streams
• Celery-backed queue if simpler

---

11) Capture observability from day one

You’ll want metrics like:

• fetch success rate
• 2xx / 3xx / 4xx / 5xx rates
• average latency
• per-domain error rates
• change rate by site
• false positive alert rate
• queue lag
• crawl freshness SLA

Also keep structured logs with:

• URL
• job ID
• worker ID
• status code
• hash values
• diff classification

---

12) Design for content type differences

Different pages need different handling:

• HTML
• JSON APIs
• PDFs
• images
• JS-heavy SPAs

For JS-heavy sites, decide whether you really need browser rendering. If yes:

• use headless Chromium only for a subset
• cache rendered snapshots
• avoid using browser rendering for everything, since it’s expensive

---

13) Use a layered change detection strategy

A practical approach:

1. Cheap filter
   • HTTP status / headers
   • hash of response body
2. Normalization
   • cleaned text hash
3. Semantic diff
   • field extraction / DOM diff
4. Deep diff only when needed
   • store detailed delta or snapshot comparison

This keeps costs manageable across thousands of URLs.

---

14) Plan for noisy pages

Some sites change constantly for reasons unrelated to content:

• rotating ads
• recommended items
• timestamps
• session IDs
• anti-bot tokens

For these, you’ll need:

• site-specific parsers
• boilerplate removal
• selective extraction
• ignore rules

---

15) Suggested minimal architecture

If you want a solid but not overcomplicated starting point:

• Scheduler DB: Postgres
• Queue: SQS / Redis / Kafka
• Workers: containerized fetchers
• Snapshot store: S3 or equivalent
• Metadata store: Postgres
• Change detection: normalized hash + optional DOM/text diff
• Monitoring: Prometheus + Grafana + logs

---

Recommended baseline workflow

For each URL:

1. check if it’s due
2. fetch with conditional headers
3. store raw response if changed or if policy requires
4. normalize content
5. compare to previous normalized version
6. classify change
7. persist version + diff summary
8. schedule next crawl based on behavior

---

If you want, I can also help you with:

• a reference architecture diagram
• a database schema for URL state/versioning
• a crawl scheduling algorithm
• or a sample implementation in Python/Scrapy/Celery/Kafka