[SSL: CERTIFICATE_VERIFY_FAILED] when using requests and https

[TL;DR] pip install pip-system-certs

If you are trying to use requests with https and get the following error, this article should help

requests.exceptions.SSLError: HTTPSConnectionPool(host='xxx', port=443):
  Max retries exceeded with url: /
    (Caused by SSLError(SSLCertVerificationError(1,
      '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
      unable to get local issuer certificate (_ssl.c:1002)')))

This error can occur when attempting to access an https endpoint using requests, for example:

requests.get("https://www.google.com")

What does the CERTIFICATE_VERIFY_FAILED error mean?

The error happens because the certificate being used by the server was not issued by a certificate authority (CA) included in the default list of trusted CAs used by the requests module.

This can happen for a few different reasons:

1. A proxy server or anti-malware/anti-virus software is scanning https connections [1].
2. The certificate the server is using is self-signed, and so not signed by one of the trusted CAs.
3. The list of trusted CAs being used by the requests module is out of date.
4. There really is a problem and either your network or the server are compromised!

Why does this only affect requests, not urllib?

The requests module uses a list of trusted Certificate Authorities packages in another package, certifi. Other packages like urllib, on Windows, use the Windows Certificate Store which may contain the missing certificate (and it the request is working with another library, it probably does!).

How to fix the problem?

If the problem is 1, 2 or 3 and the server certificate is trusted by Windows then the easiest thing to do is to make the requests package use the Windows Certificate Store and not the ones from the certifi package.

To do that, install the "pip-system-certs" package from PyPI - https://pypi.org/project/pip-system-certs/

pip install pip-system-certs

Updating the certifi package to the latest version may fix the problem if the cause is 2, but even then the above will also work.

If the certificate is not trusted by Windows, you will either need to install the server certificate and let requests use the certificates from Windows as above, or tell requests to use a different CA certificates file.

To trust only the exact certificate being used by the server (for example, if your server certificate is self signed) without installing it, download it and pass verify="/path/to/cert.pem" to the requests method, where cert.pem is the server certificate.

Footnotes

[1] Some proxy servers, anti-malware and anti-virus software will scan https requests. This is like a man-in-the-middle attack, except in this case you trust the man in the middle! The proxy server will use its own certificate and so if that certificate is not trusted then the SSL verification will fail. This will be installed on your PC, but will not be in the cacerts file that the requests package uses which is why the verification fails when using requests and works otherwise.  It might even work when running Python on its own and fail when calling the same Python code from Excel (or another application with Python embedded), it will depend how the proxy server is configured.