Prompt

How do I detect changes on websites with a crawler?

Technology / Cloud Infrastructure2 observationsLast seen Oct 14, 2026

Latest observation

Oct 14, 2026OpenAI APIWeb search: off

To detect changes on websites with a crawler, the usual pattern is:

  1. Crawl the page
  2. Extract the content you care about
  3. Normalize it
  4. Compare it to a previously saved version
  5. Trigger an alert if it changed

Common approaches

1) Hash the page or extracted content

  • Fetch the HTML.
  • Strip irrelevant parts like timestamps, ads, tracking scripts, and whitespace.
  • Compute a hash (e.g., SHA-256).
  • Store the hash.
  • On the next crawl, recompute and compare.

Good for:

  • Simple change detection
  • Large-scale monitoring

Limitation:

  • Any minor HTML change can cause a “change,” even if the visible content is the same

2) Extract structured fields and compare them

Instead of comparing the whole page, extract only the fields you care about:

  • Title
  • Price
  • Availability
  • Product description
  • Article text

Then compare those values against the previous crawl.

Good for:

  • E-commerce pages
  • News pages
  • Listings

3) Use DOM/text diffing

  • Parse the page into a DOM.
  • Remove dynamic elements.
  • Compare the cleaned DOM or visible text.
  • Optionally generate a diff to see what changed.

Good for:

  • Understanding what changed
  • More robust than raw HTML comparison

4) Screenshot comparison

Render the page in a headless browser and compare screenshots pixel-by-pixel or with image diff tools.

Good for:

  • Pages heavily built with JavaScript
  • Visual changes

Limitation:

  • More expensive and sensitive to small layout shifts

Recommended crawler workflow

Step 1: Fetch

Use:

  • requests / httpx for static pages
  • Playwright / Puppeteer / Selenium for JS-heavy pages

Step 2: Clean

Remove:

  • <script>, <style>
  • navigation, footer, sidebar if irrelevant
  • timestamps, session IDs, random tokens
  • whitespace differences

Step 3: Extract

Get:

  • full visible text, or
  • specific selectors/XPath/CSS paths

Step 4: Normalize

Examples:

  • lowercase text
  • collapse whitespace
  • sort key/value pairs
  • normalize dates/numbers
  • convert relative URLs to absolute URLs if needed

Step 5: Store snapshot

Save:

  • content hash
  • extracted fields
  • crawl timestamp
  • URL
  • optionally raw HTML for debugging

Step 6: Compare

If new normalized content differs from the stored one:

  • mark as changed
  • save the new version
  • notify via email/webhook/queue

Example logic

Pseudo-code:

content = fetch(url)
cleaned = extract_visible_text(content)
normalized = normalize(cleaned)
new_hash = sha256(normalized.encode()).hexdigest()

old_hash = db.get_hash(url)

if old_hash != new_hash:
    db.save_hash(url, new_hash)
    alert_change(url)

Best practices

  • Ignore unstable parts of pages
  • Use selectors for important data instead of whole-page comparison
  • Respect rate limits and robots.txt
  • Retry and handle errors
  • Use versioning so you can inspect historical changes
  • Consider caching ETags/Last-Modified headers to reduce unnecessary downloads
  • Use headless browsers only when needed because they’re slower and cost more

Choosing the right method

  • Whole-page hash: fastest, simplest
  • Field extraction: best for accurate monitoring
  • Text/DOM diff: best for debugging changes
  • Screenshot diff: best for visual changes or JS apps

If you want, I can show you:

  • a Python example using requests + BeautifulSoup
  • a Playwright example for JS-rendered sites
  • or a change-detection architecture for a production crawler.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (2 observations for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.