Prompt
How do I crawl millions of pages without getting blocked?
Latest observation
Nov 10, 2026
OpenAI APIWeb search: off
Crawling at that scale is as much an operations and compliance problem as a technical one. The safest way to avoid blocks is to respect the site’s rules, crawl politely, and reduce load—not to try to “beat” defenses.
Do this instead of trying to evade blocks
1) Get permission and use the right access path
- Prefer an official API, data export, feed, sitemap, or partner access.
- Check the site’s robots.txt and Terms of Service.
- If you need large-scale access, contact the site and ask for rate limits or whitelisting.
2) Crawl politely
- Use a clear, truthful User-Agent identifying your bot and contact email.
- Keep concurrency low per host.
- Add delays/jitter between requests.
- Implement exponential backoff on 429/503 and stop when the site asks you to slow down.
- Avoid hammering pages that error repeatedly.
3) Fetch less
- Don’t refetch content you already have:
- Use ETags
- Use If-Modified-Since
- Honor cache headers
- Deduplicate URLs before fetching.
- Crawl only the URLs you actually need.
4) Be a good network citizen
- Separate your crawler into:
- URL discovery
- scheduling
- fetching
- parsing
- storage
- Rate-limit per domain, not just globally.
- Spread requests over time rather than spikes.
5) Handle blocks transparently
If you see:
- 403: likely forbidden; stop and investigate access/permission.
- 429: too many requests; slow down significantly.
- 503 with retry headers: back off and retry later.
- CAPTCHAs / login walls: don’t try to bypass them; use authorized access.
Practical architecture for millions of pages
- Frontier queue: prioritized URL queue with per-host budgets.
- Per-host token bucket: e.g. 1–2 concurrent requests/host, low req/min.
- Retries with capped exponential backoff.
- Caching layer: avoid re-downloading unchanged pages.
- Monitoring:
- status code rates
- request latency
- error spikes
- per-host throttling
- Kill switch to pause if a site starts failing.
Simple policy example
- Start at 1 request/second per host.
- If 2xx is stable and latency is low, increase slowly.
- If 429/503 rises above a threshold, cut rate by 50–80%.
- Never increase concurrency aggressively on a new site.
If you want long-term scale
- Negotiate access.
- Use sitemaps and incremental crawling.
- Use change detection to revisit only updated pages.
- Consider distributed crawling with strict per-host limits.
If you want, I can help you design a polite large-scale crawler architecture or provide a sample rate-limited crawler in Python/Go that respects robots.txt and backoff.