How can I compile Python 3.6.2 on macOS with openSSL from homebrew?

Question

I'm trying to compile Python 3.6.2 on macOS 10.11 according to the instructions on https://bugs.python.org/issue29095.

I've used homebrew to install openSSL to the standard location and then added LDFLAGS, CFLAGS, and CPPFLAGS to my env:

$ printenv | grep FLAGS
LDFLAGS=/usr/local/Cellar/openssl/1.0.2l/lib/
CFLAGS=-I/usr/local/Cellar/openssl/1.0.2l/include/openssl
CPPFLAGS=-I/usr/local/Cellar/openssl/1.0.2l/include/openssl

Then in that same shell, I compile Python to the custom location required for my use case:

$ sudo ./configure --prefix=/oebuild/python/python-3.6.1
$ sudo make
$ sudo make install

However, the SSL module doesn't get built. The build log says this:

Python build finished successfully! The necessary bits to build these optional modules were not found: _gdbm _ssl ossaudiodev
spwd
To find the necessary bits, look in setup.py in detect_modules() for the module's name.

[How do I compile Python 3.4 with custom OpenSSL?](http://stackoverflow.com/q/23548188/608639), [Building Python with SSL support in non-standard location](http://stackoverflow.com/q/5937337/608639), [Building python with openssl support](http://unix.stackexchange.com/q/254974/56041), etc. — jww, Sep 27 '17 at 23:59
And some others... [How do I install pyOpenSSL on Mac OS X?](http://stackoverflow.com/q/14361569), [Updating openssl in python 2.7](http://stackoverflow.com/q/18752409), [Python referencing old SSL version](http://stackoverflow.com/q/24323858), [Python and OpenSSL version reference issue on OS X](http://stackoverflow.com/q/37690054), [Python 3.3 and Installing PyOpenSSL on a Mac](http://stackoverflow.com/q/21899573), [Using Python with homebrew on OS X](http://stackoverflow.com/q/25441252), etc... — jww, Sep 28 '17 at 00:00
And some more... [Homebrew refusing to link OpenSSL](http://stackoverflow.com/q/38670295), [Update OpenSSL on OS X with Homebrew](http://stackoverflow.com/q/15185661), [How to install latest version of openssl Mac OS X El Capitan](http://stackoverflow.com/q/35129977), [How to upgrade OpenSSL in OS X?](http://apple.stackexchange.com/q/126830), [Openssl installation using HomeBrew fails](http://superuser.com/q/486389), etc. — jww, Sep 28 '17 at 00:00

iLikeDirt · Accepted Answer · 2017-10-11T21:25:31.417

None of the previous answers I found earlier worked for me, but I did eventually figure this out with the help of another answer not mentioned earlier. Here was the actual fix: https://stackoverflow.com/a/20740964/2934226

Basically, CPPFLAGS and LDFLAGS can't be set in the environment; you need to set them alongside the configure command, like this:

./configure CPPFLAGS="-I[openSSL install location]/include" LDFLAGS="-L[openSSL install location]/lib" [other flags here]

And then after compiling and installing, it worked!

$ python3 -c "import ssl; print(ssl.OPENSSL_VERSION)"
OpenSSL 1.0.2l  25 May 2017

Here are the things that didn't work, and why:

How do I compile Python 3.4 with custom OpenSSL? doesn't help because you can't set LDFLAGS, CFLAGS, or CPPFLAGS in the environment; setup.py doesn't pass them along to the actual compilation steps. And even if setting LD_LIBRARY_PATH might work, you don't want to do that because it's dangerous (see http://xahlee.info/UnixResource_dir/_/ldpath.html). Finally, --with-ssl isn't a valid configure argument, and the patch listed there to add it doesn't apply cleanly.

Homebrew refusing to link OpenSSL doesn't apply when you're trying to build something from source, rather than trying to get an an already-compiled dylib to find a relocated library. Furthermore, making symlinks in /usr/local is dangerous, and can cause programs to compile against the newer headers but use the older system binaries.

how to include ssl with python build on MacOS doesn't work properly. Editing setup.py to add the lib and include directories for where you've installed my openSSL partially works, and lets you compile in SSL support. Alas, they aren't importable because the old version is still getting used:

Following modules built successfully but were removed because they could not be imported:
_hashlib              _ssl

[...]

building '_ssl' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -I/usr/local/opt/openssl/include/ -I./Include -I/oebuild/python/python-3.6.1/include -I. -I/usr/local/include -I/oebuild/python/src/Python-3.6.1/Include -I/oebuild/python/src/Python-3.6.1 -c /oebuild/python/src/Python-3.6.1/Modules/_ssl.c -o build/temp.macosx-10.11-x86_64-3.6/oebuild/python/src/Python-3.6.1/Modules/_ssl.o
gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.11-x86_64-3.6/oebuild/python/src/Python-3.6.1/Modules/_ssl.o -L/oebuild/python/python-3.6.1/lib -L/usr/local/lib -lssl -lcrypto -o build/lib.macosx-10.11-x86_64-3.6/_ssl.cpython-36m-darwin.so
building '_hashlib' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -I/usr/local/opt/openssl/include/ -I./Include -I/oebuild/python/python-3.6.1/include -I. -I/usr/local/include -I/oebuild/python/src/Python-3.6.1/Include -I/oebuild/python/src/Python-3.6.1 -c /oebuild/python/src/Python-3.6.1/Modules/_hashopenssl.c -o build/temp.macosx-10.11-x86_64-3.6/oebuild/python/src/Python-3.6.1/Modules/_hashopenssl.o
gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.11-x86_64-3.6/oebuild/python/src/Python-3.6.1/Modules/_hashopenssl.o -L/oebuild/python/python-3.6.1/lib -L/usr/local/lib -lssl -lcrypto -o build/lib.macosx-10.11-x86_64-3.6/_hashlib.cpython-36m-darwin.so
*** WARNING: renaming "_ssl" since importing it failed: dlopen(build/lib.macosx-10.11-x86_64-3.6/_ssl.cpython-36m-darwin.so, 2): Symbol not found: _CRYPTO_THREADID_set_callback
  Referenced from: build/lib.macosx-10.11-x86_64-3.6/_ssl.cpython-36m-darwin.so
  Expected in: flat namespace
 in build/lib.macosx-10.11-x86_64-3.6/_ssl.cpython-36m-darwin.so
*** WARNING: renaming "_hashlib" since importing it failed: dlopen(build/lib.macosx-10.11-x86_64-3.6/_hashlib.cpython-36m-darwin.so, 2): Symbol not found: _HMAC_CTX_copy
  Referenced from: build/lib.macosx-10.11-x86_64-3.6/_hashlib.cpython-36m-darwin.so
  Expected in: flat namespace
 in build/lib.macosx-10.11-x86_64-3.6/_hashlib.cpython-36m-darwin.so

otool -L shows the problem:

$ otool -L build/lib.macosx-10.11-x86_64-3.6/_ssl.cpython-36m-darwin_failed.so 
build/lib.macosx-10.11-x86_64-3.6/_ssl.cpython-36m-darwin_failed.so:
    /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
    /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

(CRYPTO_THREADID was introduced in version 1.0.0, according to https://wiki.openssl.org/index.php/Manual:Threads(3)#HISTORY)

Note that you can get the "openSSL install location" with `brew --prefix openssl`, so your command line can be `./configure CPPFLAGS="-I$(brew --prefix openssl)/include" LDFLAGS="-L$(brew --prefix openssl)/lib" ` — Kristopher Johnson, Apr 13 '18 at 11:28
This works not just for python3 installs, but also for python2. (I tried with 2.7.6) — Chris Reyes - he-him, May 10 '18 at 17:17

Guido van Rossum · Answer 2 · 2022-01-07T22:43:55.337

14

No answer above worked for me, for Python 3.8 (currently in beta).

Instead, what worked for me (July 2019, updated January 2022):

brew install openssl
./configure --with-openssl=$(brew --prefix openssl)

edited Jan 07 '22 at 22:43

answered Jul 17 '19 at 03:25

Guido van Rossum

16,690
3
46
49

This worked for me; though on M1 it took the following change: `./configure --with-openssl=$(brew --prefix openssl)` – user14717 Jan 06 '22 at 21:44
Thanks, updated my answer. – Guido van Rossum Jan 07 '22 at 22:44

score 3 · Answer 3 · answered Oct 25 '18 at 19:52

3

I had success like this.

  brew update
  brew install openssl

  # these were suggested by brew itself.

  export LDFLAGS="-L/usr/local/opt/openssl/lib"
  export CPPFLAGS="-I/usr/local/opt/openssl/include"
  export PKG_CONFIG_PATH="/usr/local/opt/openssl/lib/pkgconfig"

  # vanilla compilation

  ./configure
  make

answered Oct 25 '18 at 19:52

Senthil Kumaran

54,681
14
94
131

Thank you - only this one worked once adding the additional `PKG_CONFIG_PATH` parameter to my `./configure` command. I've written a shell wrapper for OSX to assist in my own builds which is available here: https://github.com/danstreeter/osx-python-builder - contributions welcome =) – Dan Streeter Feb 08 '20 at 15:06

score 2 · Answer 4 · answered May 29 '20 at 23:19

2

Been struggling with building from pyenv all day so figured I'd contribute the method for doing it in 2020:

brew install rbenv/tap/openssl@1.0
PYTHON_BUILD_HOMEBREW_OPENSSL_FORMULA=openssl@1.0 pyenv install 3.6.2

or in my case, 3.4.10 (what I tested this with)

I know this question isn't for pyenv specifically but I found this thread looking for help with pyenv and I'm sure others have too.

Note that this version of openssl is (currently) 1.0.2t

answered May 29 '20 at 23:19

Sweet Sheep

326
1
6

Thanks for this answer, as it's really the easiest solution for users of both `brew` and `pyenv`. However, don't forget to also check `$CONFIGURE_OPTS` and unset or override if necessary (`CONFIGURE_OPTS=""` or `CONFIGURE_OPTS="--with-openssl=$(brew --prefix openssl@1.0)"`) at the start of the second command above. Mine was set in `.zshrc` to specify `openssl@1.1` in order to steer `rbenv` toward using `openssl@1.1`, and I needed to override it to get older Pythons built using `pyenv`. – Chris Dec 07 '20 at 19:17
This worked for me, but with OpenSSL v3. I'm on an intel Mac. ``` PYTHON_BUILD_HOMEBREW_OPENSSL_FORMULA=openssl@3 pyenv install 3.6 ``` – Craig Blaszczyk Dec 12 '22 at 12:11

How can I compile Python 3.6.2 on macOS with openSSL from homebrew?

4 Answers4

Linked