How can I rotate IP addresses while scraping websites?

To rotate IPs while scraping, the usual approach is to send requests through a pool of proxies so each request, or batch of requests, appears to come from a different IP.

Common methods

1) Proxy rotation

Use a list of proxies and cycle through them:

• Datacenter proxies: cheaper, faster, but easier to detect
• Residential proxies: more expensive, often less blocked
• Mobile proxies: expensive, sometimes highest success rate

You can rotate:

• per request
• per session
• after a certain number of requests
• on failure/block

2) Proxy providers / rotating proxy gateways

Instead of managing individual proxies yourself, use a service that gives you:

• one endpoint
• automatic IP rotation
• retries and geo-targeting

This is the simplest operationally.

3) Multiple VPN exit nodes

Possible, but usually not ideal for scraping:

• slower
• harder to automate
• less scalable than proxies

---

Basic Python example with requests

import requests
import random

proxies_list = [
    "http://user:pass@proxy1.example.com:8000",
    "http://user:pass@proxy2.example.com:8000",
    "http://user:pass@proxy3.example.com:8000",
]

def fetch(url):
    proxy = random.choice(proxies_list)
    proxies = {
        "http": proxy,
        "https": proxy,
    }
    r = requests.get(url, proxies=proxies, timeout=15)
    return r.text

html = fetch("https://example.com")
print(html[:500])

---

Using a session and rotating periodically

import requests
import random

proxies_list = [
    "http://user:pass@proxy1.example.com:8000",
    "http://user:pass@proxy2.example.com:8000",
]

session = requests.Session()

for i in range(10):
    proxy = random.choice(proxies_list)
    session.proxies = {"http": proxy, "https": proxy}
    response = session.get("https://example.com", timeout=15)
    print(i, response.status_code)

---

Good practices

• Rotate user agents too
• Respect robots.txt and site terms
• Add delays and backoff
• Avoid high request rates
• Reuse sessions when you need cookies/login
• Detect bans via status codes like 403, 429, or CAPTCHA pages

---

Important caveat

Rotating IPs does not guarantee access. Many sites also fingerprint:

• headers
• cookies
• TLS/client behavior
• browser fingerprints
• request patterns

So you often need a combination of:

• proxy rotation
• rate limiting
• realistic headers
• session management
• browser automation when needed

If you want, I can show:

1. a requests-based proxy rotator,
2. a Scrapy proxy middleware, or
3. a Playwright/Selenium setup with rotating proxies.