How to Set Up Proxies with Selenium (Configuration Tutorial)

Selenium is one of the most popular open-source tools for automating web browsers. It allows you to control browsers programmatically for tasks like web scraping, testing, and automation. However, Selenium doesn't hide your IP address or rotate IPs like proxies do. Using proxies with Selenium provides many advantages for web scraping and automation.

This comprehensive guide will teach you how to configure and integrate residential and datacenter proxies with Selenium on Windows, macOS, and Linux. We'll cover setting up both free and paid proxy services, writing the necessary code in Python and Java, troubleshooting issues, maximizing performance, and more.

Why Use Proxies with Selenium?

Here are some of the main benefits of using proxies with Selenium:

• Avoid IP blocking – Many sites block scraper IPs, so proxies enable you to rotate IPs and appear from new locations.
• Obscure identity – Hiding your real IP and location info with proxies helps you scrape more anonymously.
• Overcome captchas – Proxies from diverse geographic locations can help you bypass captcha protections.
• Geo-target content– Get region-specific content by routing Selenium through country-specific proxy IPs.
• Increase scrape speed – Multi-threaded scrapes using proxies spread load and boost throughput.
• Hide headless browser – Proxies hide the fact you're using a headless browser for increased anonymity.

Now let's go through how to leverage these benefits by integrating some top proxy services with Selenium on each major OS.

Proxy Service Options

Here are some of the best paid proxy providers to use with Selenium:

• BrightData – Offers reliable residential proxies starting at $10.5/GB. Excellent region targeting.
• Smartproxy – Residential proxies starting at $14/month. Backconnect proxies rotate IPs per request.
• Proxy-Seller – Budget residential proxies starting at $10 per GB. Lots of location options.
• Soax – Rotating residential proxies starting at $99/month for 15 GB of traffic. High anonymity.

Most of these providers also offer datacenter proxy plans. Residential proxies often provide higher anonymity, while datacenter proxies are faster. Let's go through how to integrate each service on different platforms.

Selenium Setup

Before integrating proxies, you'll first need to set up Selenium if you don't already have it installed:

Windows

1. Install Python 3.6+ or Java 8+
2. Run pip install selenium for Python or add the Maven dependency for Java
3. Use webdriver-manager for Python or webdrivermanager for Java to auto-install the ChromeDriver

macOS

1. Install Homebrew if you don't have it
2. Run brew install python to install Python 3
3. Install Selenium with pip3 install selenium
4. Use webdriver-manager to auto-install the ChromeDriver

Linux

1. Use your package manager to install Python 3 or Java 8+
2. Run pip install selenium for Python or add the Maven dependency for Java
3. Install chromedriver or geckodriver for Chrome or Firefox control

Now we're ready to integrate some proxies!

BrightData Proxy Setup

BrightData provides reliable residential proxies starting at $10.5/GB. Here's how to integrate them with Selenium on each OS:

Windows

Python

from seleniumwire import webdriver 

BRIGHTDATA_PROXY = "username:[email protected]:22225"

options = {
    "proxy": {
        "http": BRIGHTDATA_PROXY,
        "https": BRIGHTDATA_PROXY
    }
}

driver = webdriver.Chrome(seleniumwire_options=options)

Java

import net.lightbody.bmp.BrowserMobProxy;

BrowserMobProxy proxy = new BrowserMobProxy("zproxy.lum-superproxy.io", 22225);
proxy.setChainedProxyAuth("username", "password");

Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
ChromeOptions options = new ChromeOptions();
options.setProxy(seleniumProxy);

WebDriver driver = new ChromeDriver(options);

To authenticate proxies with Selenium Grid for distributed scraping, provide the seGrid url instead of localhost when initializing the RemoteWebDriver.

macOS

The setup is the same as Windows, just install the ChromeDriver with brew install chromedriver or GeckoDriver with brew install geckodriver.

Linux

The Python and Java examples above work the same for Linux. Just change the driver initialization to use ChromeDriver() or FirefoxDriver() depending on your browser.

Make sure the driver version matches your browser version to avoid errors. You can also use driver.set_page_load_timeout(30) to set longer page load timeouts when working with proxies.

Smartproxy Setup

Smartproxy offers backconnect residential proxies starting at $14/month. Here's how to integrate them:

Python

from selenium import webdriver

SMARTPROXY_USER = "username"
SMARTPROXY_PASS = "password"  

manifest_json = """  
{
  "version": "1.0.0",
  "manifest_version": 2,
  "permissions": [
    "proxy",
    "tabs",
    "unlimitedStorage",
    "storage",
    "<all_urls>",
    "webRequest",
    "webRequestBlocking"
  ],
  "background": {
    "scripts": ["background.js"]
  },
  "minimum_chrome_version":"22.0.0"
}  
"""

background_js = """
var config = {
    mode: "fixed_servers",
    rules: {
      singleProxy: {
        scheme: "http",
        host: "%s",
        port: parseInt(%s) 
      },
      bypassList: ["foobar.com"] 
    }
  };

chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});  

function callbackFn(details) {
  return {
      authCredentials: {
        username: "%s",
        password: "%s"
      }
  };
}

chrome.webRequest.onAuthRequired.addListener(
        callbackFn,
        {urls: ["<all_urls>"]},
        ['blocking']
);
""" % (SMARTPROXY_HOST, SMARTPROXY_PORT, SMARTPROXY_USER, SMARTPROXY_PASS)

pluginfile = 'proxy_auth_plugin.zip'

with zipfile.ZipFile(pluginfile, 'w') as zp:
    zp.writestr("manifest.json", manifest_json)  
    zp.writestr("background.js", background_js)

options = webdriver.ChromeOptions() 
options.add_extension(pluginfile)

driver = webdriver.Chrome(options=options)

Java

import org.openqa.selenium.Proxy;

Proxy proxy = new Proxy();
proxy.setHttpProxy(SMARTPROXY_HOST + ":" + SMARTPROXY_PORT); 
proxy.setSslProxy(SMARTPROXY_HOST + ":" + SMARTPROXY_PORT);

DesiredCapabilities caps = new DesiredCapabilities();
caps.setCapability("proxy", proxy);

driver = new ChromeDriver(caps);   

//Authenticate after driver initialized
driver.get("http://username:[email protected]/authenticate");

To reuse proxies across sessions, you can export them to a JSON file using Smartproxy's API. Then load the JSON into your script to assign proxies directly.

Proxy-Seller Setup

Proxy-Seller has affordable residential proxies starting at $10 per GB. Here's how to integrate them:

Python

from seleniumwire import webdriver

PROXY_HOST = "proxyseller.com"
PROXY_PORT = "8000"
PROXY_USER = "username"
PROXY_PASS = "password"   

proxy_options = {
  "proxy": {
    "http": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}",
    "https": f"http://{PROXY_USER}:{PROXY_PASS}@{PROXY_HOST}:{PROXY_PORT}"
  }
}

driver = webdriver.Chrome(seleniumwire_options=proxy_options)

Java

import net.lightbody.bmp.BrowserMobProxy;

BrowserMobProxy proxy = new BrowserMobProxy(PROXY_HOST, PROXY_PORT);
proxy.setChainedProxyAuth(PROXY_USER, PROXY_PASS);   

Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);  
ChromeOptions options = new ChromeOptions();
options.setProxy(seleniumProxy);

WebDriver driver = new ChromeDriver(options);

Proxy Seller supports proxy whitelisting if you want to target certain sites. You can also leverage their API to implement regular proxy rotation.

Soax Setup

Soax provides fast-rotating residential proxies starting at $149/month for 10 GB of traffic:

Python

from seleniumwire import webdriver

SOAX_PROXY = "http://customer-us-pr.soax.com:8000"  
SOAX_USER = "username"
SOAX_PASS = "password"   

options = {
  "proxy": {
    "http": SOAX_PROXY,
    "https": SOAX_PROXY,  
    "no_proxy": "localhost,127.0.0.1"     
  }
  "username": SOAX_USER,
  "password": SOAX_PASS      
}

driver = webdriver.Chrome(seleniumwire_options=options)

Java

import net.lightbody.bmp.BrowserMobProxy;

BrowserMobProxy proxy = new BrowserMobProxy("customer-us-pr.soax.com", 8000);
proxy.setChainedProxyAuth(SOAX_USER, SOAX_PASS);   

Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);
ChromeOptions options = new ChromeOptions();
options.setProxy(seleniumProxy);  

WebDriver driver = new ChromeDriver(options);

Soax proxies automatically rotate, providing high anonimity. Use their portal to monitor proxy status and bans.

Advanced Proxy Techniques

Now that you know how to configure basic proxy authentication, here are some more advanced tips:

• Generate country-specific proxies by modifying the hostname (e.g. customer-fr-pr.soax.com for France).
• Export and reuse proxies using Proxy Manager or provider APIs to implement regular proxy rotation.
• Use concurrent threads and multiple proxies to maximize scraping speed.
• Set up alerting to detect banned IPs or proxy failures using service monitoring APIs.
• Use BrowserMob Proxy for more advanced HTTPS intercepting and manipulation.
• Leverage headless browsing and browser fingerprint randomization for added anonymity.
• Create proxy whitelists to target specific sites and bypass others.
• Mimic human browsing behaviors like mouse movements and scrolling when using proxies.
• Integrate proxies into CI/CD pipelines for scalable, reliable scraping.
• Containerize proxies and Selenium for easy deployment across environments.
• Implement a robust proxy pool with failover for maximum uptime.

Make sure to read the documentation for your chosen proxy provider to take advantage of all available advanced features.

Troubleshooting Common Issues

When integrating proxies and Selenium, you may run into issues like these:

• SSL errors – Export the proxy CA certificate and import it to avoid invalid cert errors.
• Captchas and blocks – Switch proxy region/IP or use proxy rotation to appear from new locations.
• Browser not using proxy – Double check your code is configuring the proxy correctly for your Selenium version.
• Slow speeds – Try a different proxy provider or region if speeds are inconsistent.
• ChromeDriver version mismatch – Reinstall ChromeDriver to match your Chrome install version.
• Connection timeouts – Optimimize page load timeouts and retry logic when using proxies.

Don't hesitate to reach out to your proxy provider's technical support if you run into any other problems getting set up.

Wrap Up

This guide covered the full setup instructions for integrating top proxy services like BrightData, Smartproxy, Proxy Seller, and Soax with Selenium on Windows, macOS, and Linux.

Proxies are invaluable for enhancing your web scraping and test automation capabilities using Selenium. Take advantage of region targeting, IP rotation, increased throughput, and greater anonymity by incorporating proxies in your scripts.

Let me know if you have any other questions! I'm happy to help explain any part of the process in more detail.