Scrapy

Scrapy plugs into helodata through its built-in HTTP_PROXY envvar support and the HttpProxyMiddleware. For per-request rotation, add scrapy-rotating-proxies.

Single static proxy

In settings.py:

HTTPPROXY_AUTH_ENCODING = "latin-1"

DOWNLOADER_MIDDLEWARES = {
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 750,
}

Then in each request:

def start_requests(self):
    proxy = "http://helo_s1a2b3c4d5e-type-res-region-us:[e-mail]:7777"
    for url in self.start_urls:
        yield scrapy.Request(url, meta={"proxy": proxy})

Or set the env vars before launching:

export HTTP_PROXY="http://helo_s1a2b3c4d5e-type-res-region-us:[e-mail]:7777"
export HTTPS_PROXY="$HTTP_PROXY"
scrapy crawl myspider

ISP rotation with scrapy-rotating-proxies

pip install scrapy-rotating-proxies

settings.py:

ROTATING_PROXY_LIST_PATH = "/path/to/isp.txt"      # one ip:port:user:pass per line
# or:
ROTATING_PROXY_LIST = [
    "http://helo_s1a2b3c4d5e:PASSWORD@198.51.100.42:8000",
    "http://helo_s1a2b3c4d5e:PASSWORD@198.51.100.43:8000",
]

DOWNLOADER_MIDDLEWARES = {
    "rotating_proxies.middlewares.RotatingProxyMiddleware": 610,
    "rotating_proxies.middlewares.BanDetectionMiddleware": 620,
}

# Optional tuning
ROTATING_PROXY_BAN_POLICY = "rotating_proxies.policy.BanDetectionPolicy"
ROTATING_PROXY_BACKOFF_BASE = 300                  # seconds

The middleware tracks failures per proxy and skips banned ones. Combine with RETRY_HTTP_CODES = [429, 502, 522, 524] for transient gateway errors.

Residential dynamic session rotation

You don't need an external list — generate sessions on the fly:

import random, string

def session_proxy(country="us"):
    sid = "".join(random.choices(string.ascii_lowercase + string.digits, k=8))
    user = f"helo_s1a2b3c4d5e-type-res-region-{country}-session-{sid}-sesstime-10"
    return f"http://{user}:[e-mail]:7777"

class MySpider(scrapy.Spider):
    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(url, meta={"proxy": session_proxy()})

Adding the helodata tag

yield scrapy.Request(
    url,
    meta={"proxy": proxy},
    headers={"X-Helodata-Tag": "campaign-acme-q2"},
)

The tag shows up in your traffic analytics — see Statistics.

Verify

def parse(self, response):
    self.logger.info("Exit IP: %s", response.headers.get(b"X-Helodata-Exit-IP", b"").decode())

Common pitfalls

• HttpProxyMiddleware disabled — Scrapy enables it by default, but custom DOWNLOADER_MIDDLEWARES dicts must keep it.
• CONCURRENT_REQUESTS too high for a small ISP batch — your batch caps at N IPs; concurrency above N just queues. Match CONCURRENT_REQUESTS_PER_DOMAIN to your pool size.