What should I use for high-volume requests to public sites?

For high-volume requests to public sites, the safest and most reliable approach is usually:

1. Use an official API if one exists

   • It’s designed for volume, stability, and clearer rate limits.
   • Less likely to break than scraping HTML.

2. Use a proper HTTP client with rate limiting and retries

   • Respect robots.txt, terms of service, and published limits.
   • Add backoff, concurrency caps, timeouts, and caching.

3. If you need scale, use rotating proxies or a data provider

   • Especially if you’re being rate-limited or geo-restricted.
   • Prefer reputable providers and stay within legal/ToS boundaries.

4. Prefer an asynchronous client or a queue-based worker system

   • Good for throughput without overwhelming the target.
   • Examples: httpx/aiohttp in Python, undici in Node.js.

5. Avoid browser automation unless necessary

   • It’s heavier, slower, and more likely to trigger anti-bot systems.
   • Use it only when the site requires real browser behavior.

If you mean tools/libraries, good defaults are:

• Python: httpx + asyncio + tenacity/custom retries
• Node.js: undici or axios + a queue like p-queue
• Distributed jobs: Celery, RabbitMQ, Kafka, or BullMQ

If you want, I can suggest a setup for:

• API access
• web scraping
• large-scale crawling
• multi-region/proxy-based requests