How many IPs do I need for large-scale web scraping?

The number of IPs you need for large-scale web scraping depends much more on request rate, target defenses, and concurrency than on “large-scale” alone.

A practical way to estimate it:

1) Start from your desired request rate

Estimate how many requests per minute/hour you want to send per target site.

2) Compare against safe per-IP limits

Many sites will tolerate only a small number of requests per IP before rate limiting, CAPTCHAs, or blocks kick in. In practice, people often design for something like:

• Low sensitivity sites: dozens to hundreds of requests/min/IP
• Moderate sensitivity sites: a few requests/min to a few dozen requests/min/IP
• High sensitivity sites: very low rates, careful pacing, and session management

3) Use this rough formula

IPs needed ≈ total requests per minute ÷ sustainable requests per minute per IP

Example:

• You need 10,000 requests/min
• You can safely do 20 requests/min/IP
• You’d need about 500 IPs

4) Also factor in:

• Concurrency: how many simultaneous connections each IP can handle
• Rotation strategy: residential, mobile, datacenter, or proxy pool
• Target diversity: one site vs many sites
• Session persistence: some scraping works better with sticky sessions
• Geo distribution: if sites serve different content by region
• Block recovery: replacement IPs for burned proxies

5) Rule of thumb

For truly large-scale scraping, the answer is usually:

• A few IPs for small/low-friction jobs
• Dozens to hundreds for medium scale
• Hundreds to thousands for aggressive, high-volume, multi-site scraping

If you want, I can help you estimate the IP count from your specific target rate, number of sites, and desired concurrency.