How to Fix Python Requests SSLError?

The SSLError in Python requests happens when trying to access a web page with an untrusted or invalid SSL certificate. This error breaks the SSL handshake between your code and the remote server, preventing any data from being transmitted securely.

In this comprehensive guide, we'll cover everything you need to know to troubleshoot and fix SSL certificate errors in Python requests.

What Causes SSLError in Python Requests?

There are a few common triggers for the SSLError in requests:

• Expired certificate: The site's SSL certificate has expired and needs to be renewed. Python requests will reject expired certs.
• Domain mismatch: The domain you're requesting doesn't match the domain on the SSL certificate. Requests will throw an error for certificate mismatches.
• Self-signed certificate: The site is using a self-signed certificate not signed by a trusted certificate authority. Python requests doesn't trust self-signed certs.
• Untrusted CA: The SSL certificate is signed by a certificate authority that isn't trusted by your Python environment.
• Problematic cipher suites: The server only supports insecure cipher suites that requests will not use.
• Revoked certificate: The SSL certificate has been revoked by the issuing CA, meaning it should no longer be trusted.
• Expired or changed intermediate certificates: Issues with intermediate certs in the chain can also trigger SSL errors.

Any of these situations will lead to requests rejecting the site's SSL certificate and throwing a SSLError. The specific error message will provide clues on the exact cause.

How to Diagnose Python Requests SSL Issues

Debugging SSL issues can be tricky. How do you track down the root cause? Here are some effective troubleshooting steps:

Check the Error Message

First look at theException message – it may provide hints on the issue:

SSLError: [SSL: TLSV1_ALERT_PROTOCOL_VERSION] tlsv1 alert protocol version (_ssl.c:590)

Here the error indicates a TLS version incompatibility issue. Compare against common SSLError messages to isolate potential causes like expired certs, domain mismatch, untrusted CAs, etc.

Enable Requests DEBUG logging

The best way to diagnose is enabling DEBUG logs in requests to see the SSL handshake:

import logging
import requests
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests.get('https://expired.badssl.com')

This prints verbose request details:

...checks for domain match
...using TLS version TLSv1.2
...adding CA /etc/ssl/certs/CustomCA.pem
...certificate expired error
...SSL handshake failed

The debug logs provide the smoking gun of exactly why SSL validation failed.

Use Online SSL Scanners

Sites like SSL Labs Server Test and OpenSSL Certificate Decoder allow testing and diagnosing SSL issues from an external client – great for corroboration.

Check OS and Python CA Stores

Many SSL problems come down to missing Certificate Authority certificates needed to validate the cert chain. Use Python's ssl.get_default_verify_paths() to check your OS and Python CA file paths. Ensure these files exist and contain necessary root and intermediate certificates.

Review Server Configuration

Check the affected server's SSL certificate, keys, protocols, ciphersuites, etc. Look for expired certificates, mixed HTTP/HTTPS content, vulnerable TLS versions, and weak ciphers. These server-side problems frequently trigger SSL handshakes errors. With some debugging legwork, you can get to the bottom of nearly any SSLError.

Quick Fix: Disabling Certificate Verification

Once you've diagnosed the issue, what's the quickest fix? You can simply disable all SSL certificate verification in requests using verify=False:

requests.get(url, verify=False)

This forces requests to skip validation and allows sites with any SSL problem to work. However, disabling verification is extremely dangerous and defeats the purpose of SSL encryption. Without verification, the connection is susceptible to man-in-the-middle attacks.

Only use this approach for public test APIs – never for transmitting user credentials or sensitive information! Now let's explore safer ways to resolve SSLErrors.

Using a Custom CA Certificate Bundle

To fix verification issues while staying secure, you can provide a custom PEM file containing any missing CA certificates required to validate the site's cert chain:

custom_ca_bundle = '/path/to/custom_ca_cert.pem'
requests.get(url, verify=custom_ca_bundle)

If the built-in requests bundle is missing a necessary intermediate or root CA, this allows augmenting the trusted certificates. For private certs, exporting the site certificate to PEM format and specifying it as the custom CA works. Always prefer using a proper chain of trust over disabling verification entirely.

Updating the Certifi CA Store

Requests relies on the certifi package and its CA bundle for verifying certificates:

import requests
print(requests.certs.where()) /usr/local/lib/python3/dist-packages/certifi/cacert.pem

Many SSLErrors arise from outdated CA certificates in certifi. You can update to the latest CA store with pip install certifi -U and potentially resolve verification issues caused by expired certificates. Automate certifi updates to stay on top of the latest certificate authorities.

Overriding Cipher Suites

Try forcing requests to use modern, secure cipher suites if you get SSL handshake errors regarding incompatible cipher suites:

from requests.packages.urllib3.util.ssl_ import DEFAULT_CIPHERS
requests.get(url, ssl_ciphers=DEFAULT_CIPHERS)

This bypasses outdated servers only supporting old insecure ciphers. Other libraries like httpx also support overriding cipher suites. Compare your current OpenSSL cipher list against the Mozilla recommended configurations to identify mismatches.

Using the SSLContext Class for Low-Level Control

For full control over the SSL handshake, create a custom SSLContext in Python's built-in ssl module:

# Create the SSL context
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.load_verify_locations('/path/to/ca_certs.pem')

# Force certificate validation  
ctx.verify_mode = ssl.CERT_REQUIRED

# Disable insecure protocols
ctx.options |= ssl.OP_NO_SSLv2
ctx.options |= ssl.OP_NO_SSLv3 

# Restrict cipher suites
ctx.set_ciphers('ECDHE+AESGCM:!RC4:!3DES')

requests.get(url, ssl_context=ctx)

This handles everything from trust stores, certificate validation, security protocols, and cipher suites. Utilize the low-level control of SSLContext to configure requests exactly as needed to fix validation issues.

Using Framework-Specific SSL Settings

If using a Python web scraping framework like Scrapy, selenium, or Beautiful Soup, use their SSL verification toggles:

Scrapy:

yield scrapy.Request(url, callback=self.parse, ssl_verify=False)

Selenium:

options = webdriver.ChromeOptions()
options.set_capability("acceptSslCerts", True)

driver = webdriver.Chrome(options=options)

Beautiful Soup:

soup = BeautifulSoup(html, "html.parser", verify=False)

These bypass SSLErrors when scraping. Only disable for public sites without credentials.

Ignoring Specific Error Codes

In rare cases, you may need to ignore certain SSL error codes to allow requests to proceed:

from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

try:
  response = requests.get(url)
except requests.exceptions.SSLError as err:  
  if err.args[0].reason == 'CERTIFICATE_VERIFY_FAILED':
    print("Ignoring error for site")
  else: 
    raise

This is highly dangerous but can be necessary as a last resort for certain sites. Know the risks before ignoring specific codes!

Automating Certificate Renewals

Many SSL issues arise from forgetful site owners not renewing expired certificates. Use Let's Encrypt clients like Certbot to automate free certificate generation and renewals. Services like SSLMate also help manage certificates at scale.

Set up monitoring to check expirations and force renewals before certificates lapse. Proper certificate lifecycle management prevents so many SSLErrors.

Leveraging SSL Troubleshooting Tools

Utilities like OpenSSL and testssl.sh are invaluable for debugging all kinds of SSL issues:

$ openssl s_client -connect expired.badssl.com:443
...
verify error:num=10:certificate has expired

Learn to use them to inspect certificates, handshake failures, mismatches, expiration dates, and more.

The Dangers of Disabling Certificate Verification

It's worth reiterating how dangerous disabling SSL certificate verification can be:

• No encryption: Disabling verification turns SSL into plain unencrypted HTTP. Sensitive data is transmitted in the clear over the internet for anyone to intercept.
• MITM attacks: There is no protection against man-in-the-middle attacks that modify traffic and steal credentials or data. An attacker can impersonate the site.
• Host forgery: Your code could get tricked into talking to a fake imposter domain rather than the real origin. This compromises security and integrity.
• User distrust: Visitors will get scary certificate warnings undermining confidence and harming your brand reputation.

Only disable verification for public test sites as a temporary workaround, never live production services transmitting private user data.

Conclusion

Robustly handling SSL certificates is a crucial skill for building bulletproof Python services. With some diligent debugging and configuration, you can squash frustrating requests SSLErrors for good.
Hopefully, this deep dive has given you lots of helpful tips and tricks for keeping those Python requests running smoothly across even the most complex SSL environments. Happy coding!