How to set up a proxy in Scrapy

What you need

The proxy is issued in the format host:port:login:password. Replace HOST, PORT, LOGIN and PASSWORD with your own values. For requests with the proxy key in meta, the built-in HttpProxyMiddleware works (the HTTPPROXY_ENABLED setting is on by default). The login and password are part of the address and must be percent-encoded.

pip install scrapy

Example

from urllib.parse import quote

import scrapy

HOST = "HOST"
PORT = "PORT"
LOGIN = "LOGIN"
PASSWORD = "PASSWORD"

# Special characters in the login and password must be encoded, otherwise the address is parsed incorrectly.
# The proxy scheme is http:// even for HTTPS sites
PROXY = f"http://{quote(LOGIN, safe='')}:{quote(PASSWORD, safe='')}@{HOST}:{PORT}"


class IpSpider(scrapy.Spider):
    name = "ip"
    custom_settings = {
        # The maximum download time of a single page, in seconds
        "DOWNLOAD_TIMEOUT": 30,
    }

    async def start(self):
        # The proxy key in meta: HttpProxyMiddleware adds Proxy-Authorization
        yield scrapy.Request("https://api.ipify.org", meta={"proxy": PROXY})

    def parse(self, response):
        # The service returns the address the request came from
        yield {"ip": response.text}

The start method is available in Scrapy 2.13 and newer. In older versions, use def start_requests(self) with the same contents instead.

If SOCKS5

Scrapy's downloader has no built-in SOCKS5 support, and we did not find a documented way. Use the proxy's HTTP port.

Common errors

1. Response 407: the login or password is wrong, or unencoded special characters remain in the address. Check the order in the issued host:port:login:password.
2. TimeoutError, TCPTimedOutError or ConnectionRefusedError: check HOST and PORT, whether the SOCKS5 port was put into HTTP mode, and increase DOWNLOAD_TIMEOUT if needed.
3. Special characters in the password: @, :, /, # and other characters break the address. Encode each value separately with quote(value, safe='').
4. The https:// scheme in the proxy address: the issued proxy works over HTTP, so write http://... even for HTTPS sites.
5. If HttpProxyMiddleware is disabled in DOWNLOADER_MIDDLEWARES, the proxy key in meta is ignored and requests go direct.