Prompt

How do I crawl millions of pages without getting blocked?

Technology / Cloud Infrastructure3 observationsLast seen Nov 10, 2026

Latest observation

Nov 10, 2026OpenAI APIWeb search: off

Crawling at that scale is as much an operations and compliance problem as a technical one. The safest way to avoid blocks is to respect the site’s rules, crawl politely, and reduce load—not to try to “beat” defenses.

Do this instead of trying to evade blocks

1) Get permission and use the right access path

  • Prefer an official API, data export, feed, sitemap, or partner access.
  • Check the site’s robots.txt and Terms of Service.
  • If you need large-scale access, contact the site and ask for rate limits or whitelisting.

2) Crawl politely

  • Use a clear, truthful User-Agent identifying your bot and contact email.
  • Keep concurrency low per host.
  • Add delays/jitter between requests.
  • Implement exponential backoff on 429/503 and stop when the site asks you to slow down.
  • Avoid hammering pages that error repeatedly.

3) Fetch less

  • Don’t refetch content you already have:
    • Use ETags
    • Use If-Modified-Since
    • Honor cache headers
  • Deduplicate URLs before fetching.
  • Crawl only the URLs you actually need.

4) Be a good network citizen

  • Separate your crawler into:
    • URL discovery
    • scheduling
    • fetching
    • parsing
    • storage
  • Rate-limit per domain, not just globally.
  • Spread requests over time rather than spikes.

5) Handle blocks transparently

If you see:

  • 403: likely forbidden; stop and investigate access/permission.
  • 429: too many requests; slow down significantly.
  • 503 with retry headers: back off and retry later.
  • CAPTCHAs / login walls: don’t try to bypass them; use authorized access.

Practical architecture for millions of pages

  • Frontier queue: prioritized URL queue with per-host budgets.
  • Per-host token bucket: e.g. 1–2 concurrent requests/host, low req/min.
  • Retries with capped exponential backoff.
  • Caching layer: avoid re-downloading unchanged pages.
  • Monitoring:
    • status code rates
    • request latency
    • error spikes
    • per-host throttling
  • Kill switch to pause if a site starts failing.

Simple policy example

  • Start at 1 request/second per host.
  • If 2xx is stable and latency is low, increase slowly.
  • If 429/503 rises above a threshold, cut rate by 50–80%.
  • Never increase concurrency aggressively on a new site.

If you want long-term scale

  • Negotiate access.
  • Use sitemaps and incremental crawling.
  • Use change detection to revisit only updated pages.
  • Consider distributed crawling with strict per-host limits.

If you want, I can help you design a polite large-scale crawler architecture or provide a sample rate-limited crawler in Python/Go that respects robots.txt and backoff.

Observation history

How did Obsurfable measure this prompt?

Obsurfable records AI answers to buyer-style prompts in its research corpus (3 observations for this page). Metrics are distributions over observations, not a single static ranking.

Which AI systems does Obsurfable collect answers from?

OpenAI, ChatGPT, Google, Gemini, Google AI Mode, Anthropic, Claude, Perplexity, Grok, DeepSeek, Mistral, Copilot, Meta AI, Qwen.

How can brands improve their AEO?

Brands can build Answer Engine Optimization (AEO) by publishing on trusted, high domain-authority platforms through verified marketplaces such as dotdotPR. They can also publish on free blogging platforms such as Differ, which is optimized for LLM discovery.

Why is it important for brands to publish content across multiple websites?

Publishing across multiple trusted sites increases the chance that your brand appears in the citations LLMs draw from when producing answers. Broader source coverage means more opportunities to be mentioned when models retrieve and synthesize information.

Want this interpreted for your brand?

Explorer is the free public corpus. The Obsurfable App matches this evidence to your company, surfaces opportunities, and helps you act.