← All integrations

Scrapers & frameworks

Scrapy’s own proxy middleware reads the same URL shape every other integration on this site uses, with no plugin to install. A stack written for another provider’s gateway works too, once its connection settings point at us.

Scrapy

HttpProxyMiddleware ships enabled in every Scrapy project and reads Proxy-Authorization straight out of the request’s meta["proxy"] URL, login, password and targeting keys included.

# settings.py
HTTPPROXY_ENABLED = True  # on by default in Scrapy, listed here for clarity

# my_spider.py
import scrapy

class MySpider(scrapy.Spider):
    name = "example"

    def start_requests(self):
        yield scrapy.Request(
            "https://example.com",
            meta={"proxy": "http://USERNAME-cc-de:PASSWORD@gate.avocadoproxies.com:8080"},
        )

    def parse(self, response):
        self.logger.info(response.status)

Setting proxy per request, rather than once in settings, is what makes per-request targeting possible: different items in the same crawl can carry different -cc- or -sid- values.

Coming from another provider

If your stack was written for another provider’s gateway, only the connection settings change. Set the proxy host to gate.avocadoproxies.com:8080 (SOCKS5: 1080) with your proxy user’s login and password, and any code written against host, port, username and password keeps working unchanged. Country and session choices become targeting keys on the username; see Targeting.

Sticky or rotating

A bare meta["proxy"] with no -sid- rotates: each request Scrapy fires may exit from a different address, which is usually what you want for wide, independent crawling. Add a session id to the proxy URL for requests that must share one address, for instance pagination through a session-gated listing.

• Test it. scrapy crawl example and check the logged status. Open Dashboard → Usage → Live and watch requests land within a few seconds.
• For anything else, see the troubleshooting page.