How do I use proxies with Scrapy?

In Scrapy, you usually use proxies by setting the meta["proxy"] value on each request.

Basic example

import scrapy

class MySpider(scrapy.Spider):
    name = "example"

    def start_requests(self):
        urls = [
            "https://example.com",
            "https://httpbin.org/ip",
        ]
        proxy = "http://user:password@proxy-host:proxy-port"

        for url in urls:
            yield scrapy.Request(url, meta={"proxy": proxy})

    def parse(self, response):
        self.logger.info(response.text)

Proxy without authentication

yield scrapy.Request(
    "https://example.com",
    meta={"proxy": "http://proxy-host:8080"}
)

Proxy with authentication

yield scrapy.Request(
    "https://example.com",
    meta={"proxy": "http://username:password@proxy-host:8080"}
)

Set proxy in a spider middleware or custom downloader middleware

If you want all requests, or requests matching certain rules, to use a proxy, a middleware is a cleaner approach.

Example middleware

class ProxyMiddleware:
    def process_request(self, request, spider):
        request.meta["proxy"] = "http://proxy-host:8080"

Then enable it in settings.py:

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.ProxyMiddleware": 350,
}

Rotating proxies

If you have a list of proxies, pick one per request:

import random

class RotateProxyMiddleware:
    def __init__(self, proxies):
        self.proxies = proxies

    @classmethod
    def from_crawler(cls, crawler):
        return cls(proxies=crawler.settings.getlist("PROXY_LIST"))

    def process_request(self, request, spider):
        request.meta["proxy"] = random.choice(self.proxies)

In settings.py:

PROXY_LIST = [
    "http://proxy1:8080",
    "http://proxy2:8080",
    "http://user:pass@proxy3:8080",
]

Notes

• Scrapy supports HTTP/HTTPS proxies via meta["proxy"].
• If the proxy requires auth, include credentials in the proxy URL.
• For SOCKS proxies, you may need an additional package such as scrapy-socks or use a custom downloader handler.

Quick troubleshooting

• Make sure the proxy URL includes the scheme: http://... or https://...
• Test the proxy with a site like https://httpbin.org/ip
• If requests still go directly, check for other middleware overriding request.meta["proxy"]

If you want, I can also show:

1. how to rotate proxies with retries,
2. how to use proxies only for certain domains,
3. how to use authenticated SOCKS proxies in Scrapy.