How can I scrape websites from different countries using residential proxies?

Scraping websites from different countries using residential proxies allows you to bypass geographical restrictions, view localized content (such as prices, languages, or availability), and minimize IP blocks.

Below is a step-by-step guide on how to configure and scrape websites across different countries using residential proxies.

---

1. Understand How Geo-Targeted Residential Proxies Work

Most residential proxy providers (e.g., Bright Data, Oxylabs, Smartproxy, Webshare, IPRoyal) allow geo-targeting directly through your proxy authentication credentials or proxy endpoints.

Instead of changing your code structure for every country, you modify the proxy username parameters (or host/port) to route requests through specific country code (ISO 3166-1 alpha-2, like us, gb, de, jp).

Standard Proxy URL Format:

http://username-country-de:password@proxy.provider.com:port

---

2. Match Request Headers to the Target Country

An IP address alone isn't always enough. Modern anti-bot and localization algorithms check HTTP headers to see if they match the IP location.

When targeting a specific country, adjust your request headers:

1. Accept-Language: Set it to the targeted language/locale (e.g., de-DE,de;q=0.9 for Germany).
2. User-Agent: Use a standard desktop browser User-Agent.
3. Timezone & Geolocation: If using headless browsers (Playwright/Selenium), override the browser's timezone and location settings.

---

3. Implementation Examples

Example A: Python (requests library) for Static Pages

This approach works for basic API calls or static HTML pages.

import requests

# 1. Define proxy credentials with country targeting (e.g., Japan = 'jp')
COUNTRY_CODE = "jp"
PROXY_USER = f"your_username-country-{COUNTRY_CODE}"
PROXY_PASS = "your_password"
PROXY_HOST = "gate.proxyprovider.com"
PROXY_PORT = "8080"

proxies = {
    "http": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
    "https": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
}

# 2. Match headers with Japanese locale
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
    "Accept-Language": "ja-JP,ja;q=0.9,en-US;q=0.8,en;q=0.7",
}

# 3. Send request
try:
    response = requests.get("https://httpbin.org/ip", proxies=proxies, headers=headers, timeout=10)
    print("Response IP Info:", response.json())
except requests.exceptions.RequestException as e:
    print("Error:", e)

---

Example B: Python (Playwright) for Dynamic/JavaScript Pages

For complex websites requiring JavaScript rendering, headless browsers like Playwright are necessary.

from playwright.sync_api import sync_playwright

def scrape_country(country_code, locale, timezone):
    with sync_playwright() as p:
        # Launch browser configured with residential proxy
        browser = p.chromium.launch(
            headless=True,
            proxy={
                "server": "http://gate.proxyprovider.com:8080",
                "username": f"your_username-country-{country_code}",
                "password": "your_password"
            }
        )
        
        # Emulate context settings matching the target region
        context = browser.new_context(
            locale=locale,
            timezone_id=timezone
        )
        
        page = context.new_page()
        
        # Navigate to target page
        page.goto("https://httpbin.org/ip")
        print(f"Content for {country_code}:", page.text_content("body"))
        
        browser.close()

# Example: Scrape as a UK user
scrape_country(country_code="gb", locale="en-GB", timezone="Europe/London")

# Example: Scrape as a German user
scrape_country(country_code="de", locale="de-DE", timezone="Europe/Berlin")

---

4. Rotating vs. Sticky Sessions

Depending on what you are scraping, choose the right proxy session management:

• Rotating Proxies (Per-Request Rotation):
  • Use case: Scraping search results, product listings, or bulk public data.
  • How it works: Every request gets a new IP from the specified country.
  • Syntax example: username-country-us-session-rand1234
• Sticky Sessions (Persistent IP):
  • Use case: E-commerce checkout flows, user logins, paginated content where session state matters.
  • How it works: Keeps the same IP address for a specified duration (e.g., 5 to 30 minutes).
  • Syntax example: Most providers let you pass a session ID tag (e.g., username-country-us-session-mySession1).

---

5. Best Practices & Troubleshooting

1. Test IP Geolocation Before Scraping:
   • Always hit an IP check endpoint (like https://httpbin.org/ip or https://ipinfo.io/json) first to verify that your proxy is routing traffic through the expected country.
2. Handle Anti-Bot Captchas and Fingerprinting:
   • Some websites block requests if browser fingerprinting detects inconsistencies (e.g., a German IP with a US-English browser timezone). Tools like Playwright-stealth or Puppeteer-extra-plugin-stealth help patch browser fingerprints.
3. Handle Proxy Retries & Rate Limits:
   • Residential IPs can occasionally disconnect or be slow. Always wrap requests in retry loops (urllib3.util.retry or Python's tenacity library) with exponential backoff.
4. Respect Legal & Ethical Boundaries:
   • Comply with target site Terms of Service, adhere to robots.txt where applicable, rate-limit your requests to avoid crashing target servers, and follow local data privacy laws (e.g., GDPR in the EU).