Prompt
How do I detect changes on websites with a crawler?
Latest observation
Oct 14, 2026
OpenAI APIWeb search: off
To detect changes on websites with a crawler, the usual pattern is:
- Crawl the page
- Extract the content you care about
- Normalize it
- Compare it to a previously saved version
- Trigger an alert if it changed
Common approaches
1) Hash the page or extracted content
- Fetch the HTML.
- Strip irrelevant parts like timestamps, ads, tracking scripts, and whitespace.
- Compute a hash (e.g., SHA-256).
- Store the hash.
- On the next crawl, recompute and compare.
Good for:
- Simple change detection
- Large-scale monitoring
Limitation:
- Any minor HTML change can cause a “change,” even if the visible content is the same
2) Extract structured fields and compare them
Instead of comparing the whole page, extract only the fields you care about:
- Title
- Price
- Availability
- Product description
- Article text
Then compare those values against the previous crawl.
Good for:
- E-commerce pages
- News pages
- Listings
3) Use DOM/text diffing
- Parse the page into a DOM.
- Remove dynamic elements.
- Compare the cleaned DOM or visible text.
- Optionally generate a diff to see what changed.
Good for:
- Understanding what changed
- More robust than raw HTML comparison
4) Screenshot comparison
Render the page in a headless browser and compare screenshots pixel-by-pixel or with image diff tools.
Good for:
- Pages heavily built with JavaScript
- Visual changes
Limitation:
- More expensive and sensitive to small layout shifts
Recommended crawler workflow
Step 1: Fetch
Use:
requests/httpxfor static pages- Playwright / Puppeteer / Selenium for JS-heavy pages
Step 2: Clean
Remove:
<script>,<style>- navigation, footer, sidebar if irrelevant
- timestamps, session IDs, random tokens
- whitespace differences
Step 3: Extract
Get:
- full visible text, or
- specific selectors/XPath/CSS paths
Step 4: Normalize
Examples:
- lowercase text
- collapse whitespace
- sort key/value pairs
- normalize dates/numbers
- convert relative URLs to absolute URLs if needed
Step 5: Store snapshot
Save:
- content hash
- extracted fields
- crawl timestamp
- URL
- optionally raw HTML for debugging
Step 6: Compare
If new normalized content differs from the stored one:
- mark as changed
- save the new version
- notify via email/webhook/queue
Example logic
Pseudo-code:
content = fetch(url)
cleaned = extract_visible_text(content)
normalized = normalize(cleaned)
new_hash = sha256(normalized.encode()).hexdigest()
old_hash = db.get_hash(url)
if old_hash != new_hash:
db.save_hash(url, new_hash)
alert_change(url)
Best practices
- Ignore unstable parts of pages
- Use selectors for important data instead of whole-page comparison
- Respect rate limits and robots.txt
- Retry and handle errors
- Use versioning so you can inspect historical changes
- Consider caching ETags/Last-Modified headers to reduce unnecessary downloads
- Use headless browsers only when needed because they’re slower and cost more
Choosing the right method
- Whole-page hash: fastest, simplest
- Field extraction: best for accurate monitoring
- Text/DOM diff: best for debugging changes
- Screenshot diff: best for visual changes or JS apps
If you want, I can show you:
- a Python example using
requests + BeautifulSoup - a Playwright example for JS-rendered sites
- or a change-detection architecture for a production crawler.