I'm building a competitor tracking system and need something better than one-off scripts

Absolutely — a competitor tracking system is a great case for moving from one-off scripts to a more structured pipeline.

A better setup usually has these parts:

1. Source registry

   • List of competitor domains, social accounts, job pages, app stores, pricing pages, etc.
   • Metadata like crawl frequency, priority, last seen changes.

2. Ingestion layer

   • Pull data from:
     • websites / pages
     • RSS or newsletters
     • social media APIs
     • job boards
     • review sites
     • app stores
     • SERP monitoring
   • Prefer APIs where possible; use scraping only when needed.

3. Change detection

   • Store snapshots of pages/content.
   • Diff against previous versions.
   • Trigger alerts only on meaningful changes:
     • pricing changes
     • feature launches
     • messaging changes
     • hiring pattern shifts
     • new integrations or case studies

4. Normalization

   • Convert all data into a common schema, e.g.:
     • source
     • competitor
     • entity_type
     • timestamp
     • content
     • change_type
     • confidence
     • tags

5. Storage

   • Raw data store for HTML/JSON/screenshots.
   • Structured DB for searchable fields and events.
   • Optional search index for fast filtering.

6. Alerting and reporting

   • Slack/email digest
   • weekly summary
   • “high priority changes only” alerts
   • trend reports over time

7. Orchestration

   • Scheduler or workflow engine to run jobs reliably.
   • Retry logic, rate limiting, deduplication, and monitoring.

A practical architecture

A common stack might be:

• Scheduler/orchestration: Airflow, Prefect, Dagster, or simple cron to start
• Scraping/API jobs: Python workers
• Queue: Redis / SQS / RabbitMQ
• Storage: Postgres + S3
• Search: Elasticsearch/OpenSearch if needed
• Alerts: Slack + email
• Observability: logs, metrics, dead-letter queue

What usually makes this “better” than scripts

• Repeatable and auditable
• Easier to add new competitors/sources
• Better deduplication and history
• Controlled crawl frequency
• Centralized alerts
• Less brittle than ad hoc jobs

If you’re early-stage

Start with:

• a competitors table
• a sources table
• a snapshots table
• a changes table
• one worker that fetches pages and computes diffs
• Slack alerts for significant diffs

Then expand from there.

If you want, I can help you design:

• a minimal database schema
• a system architecture
• or a Python implementation plan for this.