Python - requests.exceptions.SSLError - dh key too small

Question

I'm scraping some internal pages using Python and requests. I've turned off SSL verifications and warnings.

requests.packages.urllib3.disable_warnings()
page = requests.get(url, verify=False)

On certain servers I receive an SSL error I can't get past.

Traceback (most recent call last):
  File "scraper.py", line 6, in <module>
    page = requests.get(url, verify=False)
  File "/cygdrive/c/Users/jfeocco/VirtualEnv/scraping/lib/python3.4/site-packages/requests/api.py", line 71, in get
    return request('get', url, params=params, **kwargs)
  File "/cygdrive/c/Users/jfeocco/VirtualEnv/scraping/lib/python3.4/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/cygdrive/c/Users/jfeocco/VirtualEnv/scraping/lib/python3.4/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/cygdrive/c/Users/jfeocco/VirtualEnv/scraping/lib/python3.4/site-packages/requests/sessions.py", line 585, in send
    r = adapter.send(request, **kwargs)
  File "/cygdrive/c/Users/jfeocco/VirtualEnv/scraping/lib/python3.4/site-packages/requests/adapters.py", line 477, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: SSL_NEGATIVE_LENGTH] dh key too small (_ssl.c:600)

This happens both in/out of Cygwin, in Windows and OSX. My research hinted at outdated OpenSSL on the server. I'm looking for a fix client side ideally.

Edit: I was able to resolve this by using a cipher set

import requests

requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += 'HIGH:!DH:!aNULL'
try:
    requests.packages.urllib3.contrib.pyopenssl.DEFAULT_SSL_CIPHER_LIST += 'HIGH:!DH:!aNULL'
except AttributeError:
    # no pyopenssl support used / needed / available
    pass

page = requests.get(url, verify=False)

What kind of fix are you looking for? If the SSL cert has a problem, do you want to use HTTP or continue using HTTPS whilst ignoring the issue? — Marcel Wilson, Jun 24 '16 at 14:56
@SteffenUllrich You're right of course. I should have stated if 'SSL' generically has a problem. — Marcel Wilson, Jun 24 '16 at 15:00
move your code solution into the answer will help others to locate solution easily. — Larry Cai, Dec 08 '16 at 10:00

score 46 · Answer 1 · edited Apr 27 '20 at 15:58

46

this is not an extra answer just try to combine the solution code from question with extra information So others can copy it directly without extra try

It is not only a DH Key issues in server side, but also lots of different libraries are mismatched in python modules.

Code segment below is used to ignore those securitry issues because it may be not able be solved in server side. For example if it is internal legacy server, no one wants to update it.

Besides the hacked string for 'HIGH:!DH:!aNULL', urllib3 module can be imported to disable the warning if it has

import requests
import urllib3

requests.packages.urllib3.disable_warnings()
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
try:
    requests.packages.urllib3.contrib.pyopenssl.util.ssl_.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'
except AttributeError:
    # no pyopenssl support used / needed / available
    pass

page = requests.get(url, verify=False)

edited Apr 27 '20 at 15:58

Qlimax

5,241
4
28
31

answered Dec 08 '16 at 13:45

Larry Cai

55,923
34
110
156

1

It's more secure to use custom HTTPAdapter that added the rule to whole Session, i.e. requests.Session().mount("https://affected.website", MySSLContextAdapter(ssl_ciphers='HIGH:!DH:!aNULL')) where that adapter creates context = ssl.create_default_context() and sets context.set_ciphers(self.ssl_ciphers) and uses it in HTTPAdapter.init_poolmanager. – frost-nzcr4 Jan 29 '20 at 07:28
2

If you're interacting with the socket directly, you can set these ciphers through the context (in Python 3.2+ and 2.7.9+): `context.set_ciphers('HIGH:!DH:!aNULL')` – Nick K9 Apr 20 '20 at 15:44
2

You saved a business with this comment.. the bank we were using for payment had messed up their ssl this saved the day for now.. giving us enough time to change and work with another bank – Amir Heshmati Jun 29 '20 at 11:48
I do not need 'verify=False'. it works. I want to say the hacked string for 'HIGH:!DH:!aNULL', does not have serious security issue for deprecation of ((Diffie Hellman)) in TLSv1.1 and TLSv1.2 – F.Tamy Jan 26 '21 at 08:34
Thanks @Larry Cai! Your tip help me to do a POST method using Databricks with REST API. – Fernando Delago Apr 03 '22 at 10:45
If you are using `httpx` you can do effectively the same thing with `httpx._config.DEFAULT_CIPHERS += ':HIGH:!DH:!aNULL'` – Scott Stafford Apr 10 '22 at 21:01

bgoeman · Answer 2 · 2020-08-11T14:10:58.323

This also worked for me:

import requests
import urllib3
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'ALL:@SECLEVEL=1'

openssl SECLEVELs documentation: https://www.openssl.org/docs/manmaster/man3/SSL_CTX_set_security_level.html

SECLEVEL=2 is the openssl default nowadays, (at least on my setup: ubuntu 20.04, openssl 1.1.1f); SECLEVEL=1 lowers the bar.

Security levels are intended to avoid the complexity of tinkering with individual ciphers.

I believe most of us mere mortals don't have in depth knowledge of the security strength/weakness of individual ciphers, I surely don't. Security levels seem a nice method to keep some control over how far you are opening the security door.

Note: I got a different SSL error, WRONG_SIGNATURE_TYPE instead of SSL_NEGATIVE_LENGTH, but the underlying issue is the same.

Error:

Traceback (most recent call last):
  [...]
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 514, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='somehost.com', port=443): Max retries exceeded with url: myurl (Caused by SSLError(SSLError(1, '[SSL: WRONG_SIGNATURE_TYPE] wrong signature type (_ssl.c:1108)')))

score 20 · Accepted Answer · edited May 23 '17 at 12:00

20

Disabling warnings or certificate validation will not help. The underlying problem is a weak DH key used by the server which can be misused in the Logjam Attack.

To work around this you need to chose a cipher which does not make any use of Diffie Hellman Key Exchange and thus is not affected by the weak DH key. And this cipher must be supported by the server. It is unknown what the server supports but you might try with the cipher AES128-SHA or a cipher set of HIGH:!DH:!aNULL

Using requests with your own cipher set is tricky. See Why does Python requests ignore the verify parameter? for an example.

edited May 23 '17 at 12:00

Community

1
1

answered Jun 24 '16 at 14:58

Steffen Ullrich

114,247
10
131
172

Thanks for your suggestion, I was able to get past these errors using the cipher set and the post provided. – Feocco Jun 24 '16 at 20:26
Not very adept at this, how can I use your latter suggestion of trying another cipher with urllib2, please have a look at my issue https://stackoverflow.com/questions/52440129/urllib2-ssl3-check-cert-and-algorithm-dh-key-too-small – steff_bdh Sep 21 '18 at 09:31
Another option (instead of telling the client to not use DH) is to configure the server to use more bits using dhparam see pointers at https://stackoverflow.com/a/64581683/ – sparrowt May 16 '22 at 08:30
@sparrowt: the OP explicitly stated: *"I'm looking for a fix client side ideally."*: – Steffen Ullrich May 16 '22 at 09:22
Indeed hence not posting this as an answer - however people will find this question who _do_ have server access, and in that case I thought it would be helpful to provide the context of how to fix it server side (as it would have been to me when I came here) rather than side-stepping it and leaving the server using an insecure DH config which other clients would then have to avoid too. – sparrowt May 16 '22 at 10:39

score 14 · Answer 4 · edited Apr 14 '20 at 01:20

14

I had the same issue.

And it was fixed by commenting

CipherString = DEFAULT@SECLEVEL=2

line in /etc/ssl/openssl.cnf .

edited Apr 14 '20 at 01:20

Yasen

4,241
1
16
25

answered Apr 14 '20 at 00:19

ghediri

141
1
4

1

What i did was to set the 2 to 1 instead of commenting the hole line and it worked too... Better some level of security than none, right? – Juan Ignacio Avendaño Huergo Oct 08 '20 at 21:54

score 4 · Answer 5 · edited Sep 22 '21 at 13:00

Someone from the requests python library's core development team has documented a recipe to keep the changes limited to one or a few servers:

https://lukasa.co.uk/2017/02/Configuring_TLS_With_Requests/

If your code interacts with multiple servers, it makes sense not to lower the security requirements of all connections because one server has a problematic configuration.

The code worked for me out of the box. That is, using my own value for CIPHERS, 'ALL:@SECLEVEL=1'.

score 4 · Answer 6 · answered May 10 '23 at 10:03

Starting from version 2 of urllib3, or version 2.30 of requests, the DEFAULT_CIPHERS attribute is removed and most proposed solutions here do not work anymore. Instead you'll have to create an SSLContext:

from urllib3.util import create_urllib3_context
from urllib3 import PoolManager

ctx = create_urllib3_context(ciphers=":HIGH:!DH:!aNULL")
http = PoolManager(ssl_context=ctx)
http.request("GET", ...)

If you're using requests you will have to subclass your own HTTPAdapter which initializes a PoolManager, and mount it into a Session:

from urllib3.util import create_urllib3_context
from urllib3 import PoolManager
from requests.adapters import HTTPAdapter
from requests import Session()

class AddedCipherAdapter(HTTPAdapter):
  def init_poolmanager(self, conntections, maxsize, block=False):
    ctx = create_urllib3_context(ciphers=":HIGH:!DH:!aNULL")
    self.poolmanager = PoolManager(
      num_pools=connections,
      maxsize=maxsize,
      block=block,
      ssl_context=ctx
    )

s = Session()
s.mount("https://example.org", AddedCipherAdapter())
s.get("https://example.org/path")

frost-nzcr4 · Answer 7 · 2022-04-06T15:47:55.570

It's may be safer not to override the default global ciphers, but instead create custom HTTPAdapter with the required ciphers in a specific session:

import ssl
from typing import Any

import requests

class ContextAdapter(requests.adapters.HTTPAdapter):
    """Allows to override the default context."""

    def __init__(self, *args: Any, **kwargs: Any) -> None:
        self.ssl_context: ssl.SSLContext|None = kwargs.pop("ssl_context", None)

        super().__init__(*args, **kwargs)

    def init_poolmanager(self, *args: Any, **kwargs: Any) -> Any:
        # See available keys in urllib3.poolmanager.SSL_KEYWORDS
        kwargs.setdefault("ssl_context", self.ssl_context)

        return super().init_poolmanager(*args, **kwargs)

then you need to create custom context, for example:

import ssl

def create_context(
    ciphers: str, minimum_version: int, verify: bool
) -> ssl.SSLContext:
    """See https://peps.python.org/pep-0543/."""
    ctx = ssl.create_default_context()

    # Allow to use untrusted certificates.
    if not verify:
        ctx.check_hostname = False
        ctx.verify_mode = ssl.CERT_NONE

    # Just for example.
    if minimum_version == ssl.TLSVersion.TLSv1:
        ctx.options &= (
            ~getattr(ssl, "OP_NO_TLSv1_3", 0)
            & ~ssl.OP_NO_TLSv1_2
            & ~ssl.OP_NO_TLSv1_1
        )
        ctx.minimum_version = minimum_version

    ctx.set_ciphers(ciphers)

    return ctx

and then you need to configure each website with custom context rules:

session = requests.Session()
session.mount(
    "https://dh.affected-website.com",
    ContextAdapter(
        ssl_context=create_context(
            ciphers="HIGH:!DH:!aNULL"
        ),
    ),
)
session.mount(
    "https://only-elliptic.modern-website.com",
    ContextAdapter(
        ssl_context=create_context(
            ciphers="ECDHE+AESGCM"
        ),
    ),
)
session.mount(
    "https://only-tls-v1.old-website.com",
    ContextAdapter(
        ssl_context=create_context(
            ciphers="DEFAULT:@SECLEVEL=1",
            minimum_version=ssl.TLSVersion.TLSv1,
        ),
    ),
)

result = session.get("https://only-tls-v1.old-website.com/object")

After reading all the answers, I can say that @bgoeman's answer is close to mine, you can follow their link to learn more.

Just be advised that if you're using [`pip-system-certs`](https://pypi.org/project/pip-system-certs/), it [hot-patches the HTTPAdapter base class](https://gitlab.com/alelec/pip-system-certs/-/blob/master/pip_system_certs/wrapt_requests.py), meaning `super().init_poolmanager` won't behave as you expect. I found this out the hard way. — David A, Feb 25 '23 at 16:42

score 2 · Answer 8 · edited Apr 21 '20 at 22:34

I will package my solution here. I had to modify the python SSL library, which was possible since I was running my code within a docker container, but it's something that probably you don't want to do.

Get the supported cipher of your server. In my case was a third party e-mail server, and I used script described list SSL/TLS cipher suite

check_supported_ciphers.sh

#!/usr/bin/env bash

# OpenSSL requires the port number.
SERVER=$1
DELAY=1
ciphers=$(openssl ciphers 'ALL:eNULL' | sed -e 's/:/ /g')

echo Obtaining cipher list from $(openssl version).

for cipher in ${ciphers[@]}
do
echo -n Testing $cipher...
result=$(echo -n | openssl s_client -cipher "$cipher" -connect $SERVER 2>&1)
if [[ "$result" =~ ":error:" ]] ; then
  error=$(echo -n $result | cut -d':' -f6)
  echo NO \($error\)
else
  if [[ "$result" =~ "Cipher is ${cipher}" || "$result" =~ "Cipher    :" ]] ; then
    echo YES
  else
    echo UNKNOWN RESPONSE
    echo $result
  fi
fi
sleep $DELAY
done

Give it permissions:

chmod +x check_supported_ciphers.sh

And execute it:

./check_supported_ciphers.sh myremoteserver.example.com | grep OK

After some seconds you will see an output similar to:

Testing AES128-SHA...YES (AES128-SHA_set_cipher_list)

So will use "AES128-SHA" as SSL cipher.

Force the error in your code:

Traceback (most recent call last): File "my_custom_script.py", line 52, in imap = IMAP4_SSL(imap_host) File "/usr/lib/python2.7/imaplib.py", line 1169, in init IMAP4.init(self, host, port) File "/usr/lib/python2.7/imaplib.py", line 174, in init self.open(host, port) File "/usr/lib/python2.7/imaplib.py", line 1181, in open self.sslobj = ssl.wrap_socket(self.sock, self.keyfile, self.certfile) File "/usr/lib/python2.7/ssl.py", line 931, in wrap_socket ciphers=ciphers) File "/usr/lib/python2.7/ssl.py", line 599, in init self.do_handshake() File "/usr/lib/python2.7/ssl.py", line 828, in do_handshake self._sslobj.do_handshake() ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:727)
Get the python SSL library path used, in this case:

/usr/lib/python2.7/ssl.py
Edit it:

cp /usr/lib/python2.7/ssl.py /usr/lib/python2.7/ssl.py.bak

vim /usr/lib/python2.7/ssl.py

And replace:

_DEFAULT_CIPHERS = (
    'ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:'
    'ECDH+AES128:DH+AES:ECDH+HIGH:DH+HIGH:RSA+AESGCM:RSA+AES:RSA+HIGH:'
    '!aNULL:!eNULL:!MD5:!3DES'
    )

By:

_DEFAULT_CIPHERS = (
    'AES128-SHA'
    )

score 2 · Answer 9 · answered Jan 03 '21 at 12:03

2

I encounter this problem afer upgrading to Ubuntu 20.04 from 18.04, following command works for me .

pip install --ignore-installed pyOpenSSL --upgrade

answered Jan 03 '21 at 12:03

Zhen

189
1
6

I have Ubuntu 20.04 but it does not help me. – F.Tamy Jan 25 '21 at 14:32
it messed up all of my existing python packages, and unable to install any package – HERAwais Aug 01 '23 at 08:20

score 2 · Answer 10 · answered Jan 08 '23 at 15:05

2

Based on the answer given by the user bgoeman, the following code, which keeps the default ciphers only adding the security level, works.

import requests
requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += '@SECLEVEL=1'

answered Jan 08 '23 at 15:05

Romualdo Caruso

47
7

score 1 · Answer 11 · answered Nov 02 '20 at 11:01

1

On CentOS 7, search for the following content in /etc/pki/tls/openssl.cnf:

[ crypto_policy ]
.include /etc/crypto-policies/back-ends/opensslcnf.config  
[ new_oids ]

Set 'ALL:@SECLEVEL=1' in /etc/crypto-policies/back-ends/opensslcnf.config.

answered Nov 02 '20 at 11:01

Avinash Kumar Ranjan

375
3
9

score 1 · Answer 12 · edited Nov 19 '21 at 01:13

1

In docker image you can add the following command in your Dockerfile to get rid of this issue:

RUN sed -i '/CipherString = DEFAULT/s/^#\?/#/' /etc/ssl/openssl.cnf

This automatically comments out the problematic CipherString line.

edited Nov 19 '21 at 01:13

tdy

36,675
19
86
83

answered Nov 18 '21 at 14:32

Erdi

45
6

score 1 · Answer 13 · answered Nov 04 '22 at 03:17

1

If you are using httpx library, with this you skip the warning:

import httpx

httpx._config.DEFAULT_CIPHERS += ":HIGH:!DH:!aNULL"

answered Nov 04 '22 at 03:17

hhsm95

301
3
4

score 0 · Answer 14 · answered Sep 21 '22 at 21:47

I had the following error:

SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:727)

I solved it(Fedora):

python2.7 -m pip uninstall requests
python2.7 -m pip uninstall pyopenssl
python2.7 -m pip install pyopenssl==yourversion
python2.7 -m pip install requests==yourversion

The order module install cause that:

requests.packages.urllib3.contrib.pyopenssl.util.ssl_.DEFAULT_CIPHERS

AttributeError "pyopenssl" in "requests.packages.urllib3.contrib" when the module did exist.

Python - requests.exceptions.SSLError - dh key too small

14 Answers14

Linked

Related