Prompt
How do I build a proxy-backed scraper for recurring data pulls?
Latest observation
To build a proxy-backed scraper for recurring data pulls, design it as a small pipeline rather than a one-off script. The key pieces are:
- Target fetcher
- Proxy layer
- Parser / normalizer
- Storage
- Scheduler / orchestrator
- Monitoring and retry logic
1) Define the scraping contract
Before coding, decide:
- What data you need
- How often you need it
- Which pages/endpoints contain it
- Expected format: HTML, JSON, CSV, API, etc.
- Change detection: full refresh vs incremental updates
- Volume: number of URLs per run and expected throughput
If the site provides an API, use it first. It’s usually more stable than scraping HTML.
2) Use proxies for reliability, not just rotation
A proxy-backed scraper usually needs one of these patterns:
- Single proxy for low-volume, stable runs
- Rotating proxy pool for scale or anti-bot sensitivity
- Residential/mobile proxies when the site is stricter
- Datacenter proxies when the site is moderate and cost matters
Best practice:
- Keep proxy health checks
- Track success/fail rates per proxy
- Rotate on timeouts, 429s, 403s, CAPTCHAs
- Avoid over-rotating if sessions matter
3) Core architecture
A practical setup looks like this:
- Scheduler: cron, Celery, Airflow, Prefect, GitHub Actions, Kubernetes CronJob
- Queue: Redis, SQS, RabbitMQ, or just a job table
- Worker(s): fetch pages using proxy
- Parser: extract fields
- Database / object store: store raw + parsed data
- Observability: logs, metrics, alerts
Flow
- Scheduler enqueues URLs or jobs
- Worker picks job
- Worker selects proxy
- Worker fetches page
- Parse + validate data
- Store result
- Mark job success/failure
- Retry with backoff if needed
4) Proxy-aware fetching strategy
Use a request wrapper that:
- chooses a proxy
- sets reasonable headers
- applies timeouts
- handles retries
- swaps proxies on failure
Example in Python
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
PROXIES = [
"http://user:pass@proxy1.example.com:8000",
"http://user:pass@proxy2.example.com:8000",
]
def fetch(url, proxy_url=None, timeout=20):
session = requests.Session()
retry = Retry(
total=3,
backoff_factor=1,
status_forcelist=[429, 500, 502, 503, 504],
allowed_methods=["GET", "POST"]
)
adapter = HTTPAdapter(max_retries=retry)
session.mount("http://", adapter)
session.mount("https://", adapter)
proxies = {
"http": proxy_url,
"https": proxy_url,
} if proxy_url else None
headers = {
"User-Agent": "Mozilla/5.0 (compatible; DataBot/1.0)",
"Accept-Language": "en-US,en;q=0.9",
}
resp = session.get(url, headers=headers, proxies=proxies, timeout=timeout)
resp.raise_for_status()
return resp.text
# Usage
html = fetch("https://example.com/data", proxy_url=PROXIES[0])
5) Make proxy selection resilient
A simple proxy pool strategy:
- Keep a list of proxies
- Mark each proxy as:
- healthy
- degraded
- dead
- Use a round-robin or weighted random selector
- Increase weight for proxies with better success rates
Track per proxy:
- success count
- failure count
- latency
- last used time
- last error type
6) Handle anti-bot defenses carefully
Common signals:
- 403/401
- 429 rate limiting
- CAPTCHA pages
- unusual redirects
- empty or placeholder content
Responses:
- slow down
- reduce concurrency
- rotate IPs less aggressively
- add jitter to delays
- use session cookies if the site expects continuity
- prefer APIs where possible
- ensure compliance with site terms and robots.txt where applicable
7) Parsing and validation
Separate fetching from parsing so you can re-parse old raw data if needed.
Typical pipeline:
- save raw response
- parse fields
- validate schema
- deduplicate
- write structured rows
Example validation checks:
- required fields present
- numeric values parse correctly
- timestamps in expected range
- URLs canonicalized
- duplicates removed
8) Store raw and structured data
Use both:
- Raw HTML/JSON in object storage or files
- Structured records in a database
Why?
- debugging
- replaying parsing logic
- diffing changes over time
- auditing failures
Common choices:
- PostgreSQL for structured data
- S3/GCS/local disk for raw snapshots
9) Scheduling recurring pulls
For recurring jobs:
- Daily/Hourly: cron or Cloud scheduler
- Complex dependencies: Airflow/Prefect
- Distributed workloads: Celery + Redis/RabbitMQ
Recommended pattern:
- run every N minutes/hours
- only fetch items changed since last run
- store a “last successful run” checkpoint
- support idempotent writes
10) Observability
At minimum, log:
- URL
- proxy used
- response status
- latency
- parse success/failure
- records extracted
- retry count
Metrics:
- success rate
- per-proxy failure rate
- median latency
- data freshness
- records per run
Alerts:
- sudden 403/429 spike
- zero records extracted
- proxy pool degradation
- run misses schedule
11) Production hardening checklist
- Timeouts everywhere
- Retries with exponential backoff
- Concurrency limits
- Proxy pool health scoring
- Raw data retention
- Idempotent storage
- Schema validation
- Logging and metrics
- Alerting on anomaly
- Respect legal/contractual restrictions
12) If the site uses JavaScript
If content is rendered client-side, use:
- Playwright or Selenium
- optionally behind proxies
- headless browser with rate limits
Use browser automation only when necessary; it’s heavier than HTTP scraping.
Playwright proxy example
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(
headless=True,
proxy={"server": "http://proxy1.example.com:8000",
"username": "user",
"password": "pass"}
)
page = browser.new_page()
page.goto("https://example.com/data", wait_until="networkidle")
content = page.content()
browser.close()
13) A good minimal stack
For a small recurring scraper:
- Python
- Requests / BeautifulSoup or lxml
- PostgreSQL
- Redis + Celery
- Cron or Prefect
- Proxy provider with health monitoring
For a more robust setup:
- Python
- Playwright for JS pages
- Celery workers
- PostgreSQL
- S3 for raw captures
- Prometheus/Grafana for metrics
14) Suggested implementation plan
- Build a normal scraper without proxies
- Add raw-response storage
- Add structured parsing and validation
- Add retry and backoff
- Add proxy rotation
- Add health scoring
- Add scheduler
- Add monitoring and alerts
- Scale concurrency gradually
If you want, I can also give you:
- a full Python project skeleton
- a proxy rotation class
- a Celery/cron-based recurring scraper template
- or a Playwright version for JS-heavy sites