I'm building a marketplace watcher that revisits pages automatically

Sounds good — I can help you design it.

A marketplace watcher that revisits pages automatically usually needs 5 parts:

1. Page scheduler

   • Decides when each page should be revisited
   • Supports intervals like every 30s, 5m, 1h, etc.

2. Fetcher

   • Loads the page content
   • Could be plain HTTP requests for static pages
   • Or browser automation like Playwright/Selenium for dynamic pages

3. Change detector

   • Compares current content with the previous version
   • Can track:
     • price changes
     • title/status changes
     • new listings
     • removed items

4. Storage

   • Saves:
     • URL
     • last fetched content/hash
     • timestamps
     • history of changes
   • A database like SQLite/Postgres is usually enough to start

5. Notifier

   • Sends alerts when something changes
   • Email, Slack, webhook, SMS, push notification, etc.

A simple architecture could look like:

• Input: list of marketplace URLs
• Worker loop:
  • fetch page
  • normalize important fields
  • compare to previous snapshot
  • if changed, store diff + notify
  • schedule next revisit

A few important design choices:

• Avoid unnecessary revisits: use backoff or adaptive intervals
• Respect robots/terms: marketplaces often restrict scraping
• Handle anti-bot protections: rate limiting, headers, login sessions, browser automation
• Normalize content: strip timestamps, ads, dynamic sections so false positives are reduced

If you want, I can help you with any of these next:

• a system design
• a Python implementation
• a database schema
• a change-detection strategy
• or a crawler scheduler example