How to compile python3 on RHEL with SSL? SSL cannot be imported

Question

I'm trying to compile python on RHEL because my current python is using an old 1.0.2k ssl version.

(test_env) [brad@reason tlscheck]$ python3 --version
Python 3.9.3
(test_env) [brad@reason tlscheck]$ python3 -c "import ssl; print(ssl.OPENSSL_VERSION)"
OpenSSL 1.0.2k-fips  26 Jan 2017
(test_env) [brad@reason tlscheck]$ openssl version
OpenSSL 1.1.1l  24 Aug 2021

I think the issue is that when I compiled 3.9.3, I had not updated my OpenSSL version. I have since updated my OpenSSL and need to use it with python. So I have downloaded the newest python 3.10, but in the make stage I get an error that it will not make with ssl. I the following message:

Following modules built successfully but were removed because they could not be imported:
_hashlib              _ssl                                     


Could not build the ssl module!
Python requires a OpenSSL 1.1.1 or newer

This is the full log of trying to compile: https://pastebin.com/36EntpFz

When I use the configure options that @tony-yip mentioned, I get the following in my configure.

checking for openssl/ssl.h in /etc/ssl... no
checking whether compiling and linking against OpenSSL works... no

I'm determining my openssl location with:

[brad@reason Python-3.10.0]$ openssl version -d
OPENSSLDIR: "/etc/ssl"

To configure, I'm using:

./configure --with-openssl="/etc/ssl"

When I look for ssl.h, I find it in /usr/include/openssl. So I linked this directory to lib in /etc/ssl, but it was no help.

[brad@reason Python-3.10.0]$ ls -l /etc/ssl
total 40
lrwxrwxrwx 1 root root    16 Jul 16  2020 certs -> ../pki/tls/certs
-rw-r--r-- 1 root root   412 Oct 12 02:53 ct_log_list.cnf
-rw-r--r-- 1 root root   412 Oct 12 02:53 ct_log_list.cnf.dist
lrwxrwxrwx 1 root root    20 Oct 18 10:22 lib -> /usr/include/openssl
drwxr-xr-x 2 root root  4096 Oct 12 02:53 misc
-rw-r--r-- 1 root root 10909 Oct 12 02:53 openssl.cnf
-rw-r--r-- 1 root root 10909 Oct 12 02:53 openssl.cnf.dist
drwxr-xr-x 2 root root  4096 Oct 12 02:53 private
[brad@reason Python-3.10.0]$ sudo find / -name ssl.h | grep include
find: ‘/tmp/.mount_jetbraAJFEnl’: Permission denied
/home/brad/Downloads/freerdp-2.0.0-rc4/winpr/include/winpr/ssl.h
/home/brad/Downloads/FreeRDP/winpr/include/winpr/ssl.h
/home/brad/Development/tlscheck/openssl-1.1.1l/include/openssl/ssl.h
/usr/include/openssl/ssl.h
/var/lib/docker/overlay2/23e6f3c164ec8939352891c99393669df4ed6e66da1e04ce84616073f08c6051/diff/usr/include/openssl/ssl.h
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/18.08/c8075e929daaffcbe5c78c9e87c0f0463d75e90d2b59c92355fa486e79c7d0e3/files/include/nss/ssl.h
/var/lib/flatpak/runtime/org.freedesktop.Sdk/x86_64/18.08/c8075e929daaffcbe5c78c9e87c0f0463d75e90d2b59c92355fa486e79c7d0e3/files/include/openssl/ssl.h
find: ‘/run/user/1000/gvfs’: Permission denied

This may be extraneous information, but my libssl.so is here:

[brad@reason Python-3.10.0]$ ls /usr/lib64 | grep ssl
libevent_openssl-2.0.so.5
libevent_openssl-2.0.so.5.1.9
libssl3.so
libssl.so
libssl.so.10
libssl.so.1.0.2k
openssl

Any thoughts on why make isn't able to include ssl, please let me know. Thanks.

https://stackoverflow.com/questions/49320993/how-to-enable-fips-mode-for-libcrypto-and-libssl-packaged-with-python/49382743#49382743 — CristiFati, Oct 12 '21 at 11:00

score 18 · Answer 1 · answered Oct 21 '21 at 14:44

18

Had a very similar problem, with openssl not working and giving the same errors with python 3.10 on centos 7. Download openssl unpack then go to that directory

./config --prefix=/usr/local/custom-openssl --openssldir=/etc/ssl
make -j1 depend
make -j8
make install_sw

Then go to the python source unpack it and run in the directory

./configure -C --with-openssl=/usr/local/custom-openssl --with-openssl-rpath=auto --prefix=/usr/local/python-3.version
make -j8
make altinstall

See also Custom OpenSSL on https://docs.python.org/3/using/unix.html.

answered Oct 21 '21 at 14:44

karel van dongen

315
1
8

This was it for me, I have been trying to fix this for days – doghousedean Feb 04 '22 at 15:02
When I used this `--with-openssl-rpath=auto`, to my surprise it actually added a RUNPATH in the .so, not an rpath. Was no problem for me, but they do behave differently, so that's something to watch out for if you care about the differences. – wim Nov 18 '22 at 21:17

score 16 · Answer 2 · answered Dec 23 '21 at 22:33

This was very helpful, thanks! What kicked me onto this was the inability to run/update pip in a venv, so there wasn't much of an alternative to figuring out how to get OpenSSL to work.

As of this week (21 December 2021), in my environment (CentOS Linux release 7.9.2009 (Core)), I was able to shortcut parts of this.

From EPEL, installing openssl11, (yes, also openssl11-devel); to set up Python 3.10 properly I needed to create that "OpenSSL Directory" which I did by:

mkdir /usr/local/openssl11

Then setting symlinks for the (newly-installed) requirements:

cd /usr/local/openssl11
ln -s /usr/lib64/openssl11 lib
ln -s /usr/include/openssl11 include

And, finally, feeding this location to the configure script:

./configure --with-openssl=/usr/local/openssl11

Thanks for the really helpful explanations of how and why this was all necessary. Hope this helps others...

This was the only way I could get pip to work (so that I could get TurboGears working.). — JasonGabler, Jan 25 '22 at 21:27
Thanks man! you saved my day! tested everything but this does the trick! — Firouziam, Feb 16 '23 at 15:05

score 5 · Answer 3 · answered Nov 29 '21 at 17:56

As the output suggest, you need to point to openssl11. The easiest way I found to do it is to change the configure file in the python 3.10 source code directory:

$ sed -i 's/PKG_CONFIG openssl /PKG_CONFIG openssl11 /g' configure
$ sudo ./configure --enable-optimizations
.
.
checking whether compiling and linking against OpenSSL works... yes
.

Then you can do as @CristiFati suggested in order to check before installing that it is configured properly:

$ ./python -c "import sys, ssl; print(\"{:s}\n{:s}\".format(sys.version, ssl.OPENSSL_VERSION))"
3.10.0 (default, Nov 29 2021, 17:48:34) [GCC 7.3.1 20180712 (Red Hat 7.3.1-13)]
OpenSSL 1.1.1g FIPS  21 Apr 2020

CristiFati · Answer 4 · 2023-02-25T23:21:41.400

1. Intro

Some references that might be (more or less) useful:

[SO]: How to enable FIPS mode for libcrypto and libssl packaged with Python? (@CristiFati's answer)
[SO]: How to include OpenSSL in Visual Studio (@CristiFati's answer) (even if it's for Win, it has some good points that apply to Nix as well)
[SO]: OpenSSL FIPS_mode_set not working in Python cryptography library (@CristiFati's answer)
[SO]: ./python: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory (@CristiFati's answer)

Working on a CentOS 7 image from Docker Hub (as it's closest to RHEL 7 (that you're having))

[root@cfati-5510-0:/work/q069539286]> uname -a
Linux cfati-5510-0 5.11.0-37-generic #41~20.04.2-Ubuntu SMP Fri Sep 24 09:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
[root@cfati-5510-0:/work/q069539286]> cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)
[root@cfati-5510-0:/work/q069539286]> 
[root@cfati-5510-0:/work/q069539286]> rpm -qa | grep openssl
openssl-libs-1.0.2k-22.el7_9.x86_64
openssl-1.0.2k-22.el7_9.x86_64
openssl-devel-1.0.2k-22.el7_9.x86_64
[root@cfati-5510-0:/work/q069539286]> 
[root@cfati-5510-0:/work/q069539286]> openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017
[root@cfati-5510-0:/work/q069539286]> which openssl
/usr/bin/openssl
[root@cfati-5510-0:/work/q069539286]> ldd /usr/bin/openssl
    linux-vdso.so.1 =>  (0x00007ffe101b7000)
    libssl.so.10 => /lib64/libssl.so.10 (0x00007f9483b8a000)
    libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f948393d000)
    libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f9483654000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f9483450000)
    libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f948321d000)
    libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f9482dba000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f9482bb6000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f94829a0000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f94825d2000)
    libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f94823c2000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f94821be000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f9481fa4000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f9481d88000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f9483dfc000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f9481b61000)
    libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f94818ff000)

2. OpenSSL considerations

As seen (also present in previous snippet), CentOS 7 (and I think RHEL 7 as well) comes with OpenSSL 1.0.2*. Since OpenSSL 1.1.1* is a minor (but kind of major) release, they are not (API / ABI) compatible, and therefore not interchangeable
v1.0.2* is de facto (main) version for CentOS 7 (RHEL 7 too), everything is built against it, changing it would break things. But it is possible to install newer versions (I used the EPEL (more on [RedHat]: What's EPEL, and how do I use it?) repo ([Pkgs.CentOS]: EPEL x86_64) which provides v1.1.1g). But they will only work around the main version. For example, almost every (important) file name in the newer version has a suffix at the end (the .exe is called openssl11) in order to avoid name clashes with main version
- Since v1.1.1g installs in the (pretty much) same location as the main version (with changed names), it won't get picked up (by default) by any build system. So I had to unpack the rpms (via RPM2Cpio) in a custom dir
- Not sure how you got v1.1.1l - probably built it yourself (or found it somewhere), so some of the next steps might not be required.
  As a side note, I built (an in house modified) OpenSSL version on a variety of OSes / CPU architecutes (RHEL 7 included) many times, but since the rpms were available, I didn't bother to do it again
Even if openssl11 (and its dependent openssl11-libs) is "up and running", openssl11-devel is also required by the Python build

[root@cfati-5510-0:/work/q069539286]> mkdir -p openssl-1.1.1g && cd openssl-1.1.1g
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ls
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> for g in /var/cache/yum/x86_64/7/epel/packages/openssl11-libs-1.1.1g-3.el7.x86_64.rpm /var/cache/yum/x86_64/7/epel/packages/openssl11-1.1.1g-3.el7.x86_64.rpm /var/cache/yum/x86_64/7/epel/packages/openssl11-devel-1.1.1g-3.el7.x86_64.rpm; do rpm2cpio ${g} | cpio -idm; done
7292 blocks
2123 blocks
8352 blocks
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ls
etc  usr
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ll ./usr/bin/
total 637
-rwxrwxrwx 1 root root    610 Mar 29  2021 make-dummy-cert
-rwxrwxrwx 1 root root 644424 Mar 29  2021 openssl11
-rwxrwxrwx 1 root root    725 Mar 29  2021 renew-dummy-cert
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ./usr/bin/openssl11 
./usr/bin/openssl11: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ll ./usr/lib64/
total 3605
drwxrwxrwx 1 root root       0 Oct 21 23:55 engines-1.1
lrwxrwxrwx 1 root root      19 Oct 21 23:55 libcrypto.so.1.1 -> libcrypto.so.1.1.1g
-rwxrwxrwx 1 root root 3082216 Mar 29  2021 libcrypto.so.1.1.1g
lrwxrwxrwx 1 root root      16 Oct 21 23:55 libssl.so.1.1 -> libssl.so.1.1.1g
-rwxrwxrwx 1 root root  603568 Mar 29  2021 libssl.so.1.1.1g
drwxrwxrwx 1 root root       0 Oct 21 23:55 openssl11
drwxrwxrwx 1 root root       0 Oct 21 23:55 pkgconfig
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ldd ./usr/bin/openssl11 
    linux-vdso.so.1 =>  (0x00007ffca03f5000)
    libssl.so.1.1 => not found
    libcrypto.so.1.1 => not found
    libz.so.1 => /lib64/libz.so.1 (0x00007f3a61e56000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f3a61c52000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3a61a36000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f3a61668000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f3a6206c000)
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> # Set LD_LIBRARY_PATH
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> LD_LIBRARY_PATH=./usr/lib64:${LD_LIBRARY_PATH} ldd ./usr/bin/openssl11
    linux-vdso.so.1 =>  (0x00007ffdf7fce000)
    libssl.so.1.1 => ./usr/lib64/libssl.so.1.1 (0x00007fe00f7a4000)
    libcrypto.so.1.1 => ./usr/lib64/libcrypto.so.1.1 (0x00007fe00f2c1000)
    libz.so.1 => /lib64/libz.so.1 (0x00007fe00f0ab000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fe00eea7000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe00ec8b000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fe00e8bd000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fe00fa34000)
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> LD_LIBRARY_PATH=./usr/lib64:${LD_LIBRARY_PATH} ./usr/bin/openssl11 version
OpenSSL 1.1.1g FIPS  21 Apr 2020
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> LD_LIBRARY_PATH=./usr/lib64:${LD_LIBRARY_PATH} ./usr/bin/openssl11 version -d
OPENSSLDIR: "/etc/pki/tls"

So, here's a working OpenSSL 1.1.1g FIPS version.

3. Python build

OpenSSL is fine from its PoV, but it needs some "small" adjustments in order for Python build to pick it up:

include and lib dirs must have the same parent (they reside lower in the dir tree, symlink them in the OpenSSL root dir)
Libraries with the 11 suffix (from earlier) must also be present, symlink them as well

[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> # OpenSSL working. Prepare for Python build.
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> # "include" and "lib" dirs must be at the same level - symlink them
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ln -s ./usr/include/openssl11 ./include
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ln -s ./usr/lib64/openssl11 ./lib
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ls . ./lib
.:
etc  include  lib  usr

./lib:
libcrypto.so  libssl.so
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> # "lib" dir must also contain $(LIBNAME).1.1 besides what's already in there ($(LIBNAME))
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ln -s libcrypto.so ./lib/libcrypto.so.1.1
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ln -s libssl.so ./lib/libssl.so.1.1
[root@cfati-5510-0:/work/q069539286/openssl-1.1.1g]> ls ./lib
libcrypto.so  libcrypto.so.1.1  libssl.so  libssl.so.1.1

Python (configure and make):

root@cfati-5510-0:/work/q069539286]> tar -zxf Python-3.10.0.tgz 
[root@cfati-5510-0:/work/q069539286]> cd Python-3.10.0
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> ./configure --with-openssl=$(pwd)/../openssl-1.1.1g --with-openssl-rpath=auto --prefix=$(pwd)/../python-3.10>../cfg.txt 2>&1
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> echo $?
0
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> LD_LIBRARY_PATH=$(pwd)/../openssl-1.1.1g/lib:${LD_LIBRARY_PATH} make -j8>../mk.txt 2>&1
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> echo $?
0

4. Test

Build was successful.
Quick tests (without make install):

[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> # Quick test
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> # --------------------------------
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> ./python -c "import sys, ssl; print(\"{:s}\n{:s}\".format(sys.version, ssl.OPENSSL_VERSION))"
3.10.0 (default, Oct 22 2021, 01:12:00) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
OpenSSL 1.1.1g FIPS  21 Apr 2020
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> # --------------------------------
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> # As opposed to "regular" Python
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> python -c "import sys, ssl; print(\"{:s}\n{:s}\".format(sys.version, ssl.OPENSSL_VERSION))"
2.7.5 (default, Nov 16 2020, 22:23:17) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
OpenSSL 1.0.2k-fips  26 Jan 2017

[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> ./python -c "import ssl; print(ssl._ssl)"
<module '_ssl' from '/work/q069539286/Python-3.10.0/build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so'>
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> ldd /work/q069539286/Python-3.10.0/build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so
    linux-vdso.so.1 =>  (0x00007ffef6b7b000)
    libssl.so.1.1 => /work/q069539286/Python-3.10.0/../openssl-1.1.1g/lib/libssl.so.1.1 (0x00007f0ab9745000)
    libcrypto.so.1.1 => /work/q069539286/Python-3.10.0/../openssl-1.1.1g/lib/libcrypto.so.1.1 (0x00007f0ab9262000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0ab9046000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f0ab8c78000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f0ab8a62000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f0ab885e000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f0ab9c05000)
[root@cfati-5510-0:/work/q069539286/Python-3.10.0]> readelf -d /work/q069539286/Python-3.10.0/build/lib.linux-x86_64-3.10/_ssl.cpython-310-x86_64-linux-gnu.so | grep ssl
 0x0000000000000001 (NEEDED)             Shared library: [libssl.so.1.1]
 0x000000000000001d (RUNPATH)            Library runpath: [/work/q069539286/Python-3.10.0/../openssl-1.1.1g/lib]

What did you get for `readelf -d _ssl.cpython-310-x86_64-linux-gnu.so` ? — wim, Nov 18 '22 at 21:20
@wim: unfortunately I don't have container and image anymore (on *Docker Desktop* (*Win*)). There's a small chance that I could still have it on my *Linux* system (dual-boot) next time I'll work on it will check. But anyway since it worked with no *LD\_LIBRARY\_PATH*, I'm expecting it to have *DT\_RUNPATH* (*DT\_RPATH*) flag set (also because of `--with-openssl-rpath=auto`). — CristiFati, Nov 18 '22 at 22:48
I got the RUNPATH which surprised me because the configure flag suggests its putting RPATH. And RPATH is what I wanted, not RUNPATH, because I _don't_ want env LD_LIBRARY_PATH to have any say here. — wim, Nov 18 '22 at 23:03
I studied some years ago this topic. In many cases I see them used interchangeably. Anyway, I think that option translates to linker's *-rpath*. — CristiFati, Nov 18 '22 at 23:26
@wim: I found the container (on my other dual boot *OS*). It's indeed *DT\_RUNPATH* (but *DT\_RPATH* is deprecated). Added to the answer. — CristiFati, Feb 25 '23 at 23:05

score 0 · Answer 5 · edited Oct 26 '21 at 19:29

Python will use the bundled ssl (witch is outdated sometimes). In order to then use OpenSSL, add flag for OpenSSL when running ./configure. For detail, run ./configure --help for more options.

  --with-openssl=DIR      root of the OpenSSL directory
  --with-openssl-rpath=[DIR|auto|no]
                          Set runtime library directory (rpath) for OpenSSL
                          libraries, no (default): don't set rpath, auto:
                          auto-detect rpath from --with-openssl and
                          pkg-config, DIR: set an explicit rpath
  --with-ssl-default-suites=[python|openssl|STRING]
                          override default cipher suites string, python: use
                          Python's preferred selection (default), openssl:
                          leave OpenSSL's defaults untouched, STRING: use a
                          custom string, python and STRING also set TLS 1.2 as
                          minimum TLS version

*Python* will **not** "*use it own bundled ssl suite*" (there's no such thing). It will rely on the default *OpenSSL* installation (if any). — CristiFati, Oct 22 '21 at 03:55

How to compile python3 on RHEL with SSL? SSL cannot be imported

5 Answers5

1. Intro

2. OpenSSL considerations

3. Python build

4. Test

Linked