I'm trying to collect competitor pricing data and need a recommendation for scraping infrastructure

If you’re collecting competitor pricing data, I’d recommend building for reliability, observability, and low block rate—not just raw throughput.

Best-practice scraping infrastructure

1) Use a modular architecture

A solid setup usually looks like:

• Scheduler / queue: decides what to fetch and when
• Fetcher layer: HTTP client or browser automation
• Parser / extractor: turns pages into structured product/price data
• Storage: database + raw HTML/snapshots
• Monitoring: success rate, ban rate, latency, change detection
• Proxy / identity management: if you need to distribute traffic across IPs

2) Choose the right fetching method

• Simple HTTP scraping: fastest and cheapest for static pages or APIs
• Headless browser automation: needed for JS-heavy sites
• Hybrid approach: start with HTTP, fall back to browser only when necessary

3) Infrastructure options

Small scale / proof of concept

• Python + Scrapy for crawling
• Requests/HTTPX for simple pages
• Playwright for dynamic pages
• PostgreSQL for structured storage
• S3-compatible object storage for raw HTML/snapshots
• Redis + Celery / RQ for task queues

Medium to large scale

• Dockerized workers on:
  • AWS ECS/Fargate
  • Kubernetes
  • GCP Cloud Run / GKE
  • Azure Container Apps / AKS
• Central queue: SQS, RabbitMQ, Kafka, or Redis Streams
• Proxy management: vendor-based rotation, geo-targeting if needed
• Metrics stack: Prometheus + Grafana, plus alerting on failure spikes

4) Data quality and change detection

For pricing data, store:

• product URL
• timestamp
• raw page snapshot
• parsed price
• currency
• availability
• seller name
• shipping cost if relevant

Also track:

• page hash
• DOM selector version
• parsing errors
• HTTP status codes

This helps when sites redesign or prices are rendered differently.

5) Anti-blocking and resilience

Use:

• rate limiting
• randomized scheduling
• exponential backoff
• retries with jitter
• user-agent rotation
• session handling / cookie persistence
• careful concurrency limits per domain

6) Compliance note

Make sure your approach complies with:

• site Terms of Service
• robots.txt where applicable
• local laws and contractual restrictions

For competitor pricing, it’s often better to prioritize publicly accessible data, and avoid bypassing access controls.

Practical recommendation

If you want a balanced stack, I’d start with:

• Scrapy + Playwright
• PostgreSQL
• S3/Blob storage for raw pages
• Redis queue
• Docker on ECS/Fargate or a small Kubernetes cluster
• Prometheus/Grafana for monitoring

That setup scales from a few hundred pages/day to a fairly large operation without overengineering.

If you want, I can also propose:

1. a low-cost MVP stack,
2. a production-grade architecture, or
3. a vendor comparison for proxies, browsers, and hosting.