19

Without being aware of it, I've been using the cryptography package for many months now without issue, but suddenly it won't build and it's brought development on my project to a halt.

It turns out cryptography is a dependency of one of the other packages in my Pipfile: apns2. As I said, it's been working fine, but something must have changed in my environment. Here is my Pipfile with just that package isolated:

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[requires]
python_version = "3.7.9"

[packages]

apns2 = "*"

[dev-packages]

This fails. I use pipenv (the latest version 2020.11.15), so when I run it now this is the output:

# pipenv install
Creating a virtualenv for this project...
Pipfile: /Users/dylan/Dev/RideHare/server/Pipfile
Using /Users/dylan/.pyenv/versions/3.7.9/bin/python3.7m (3.7.9) to create virtualenv...
⠏ Creating virtual environment...created virtual environment CPython3.7.9.final.0-64 in 502ms
  creator CPython3Posix(dest=/Users/dylan/.local/share/virtualenvs/server-2uc2X-TS, clear=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/dylan/Library/Application Support/virtualenv)
    added seed packages: pip==20.2.2, setuptools==51.0.0, wheel==0.35.1
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /Users/dylan/.local/share/virtualenvs/server-2uc2X-TS
Installing dependencies from Pipfile.lock (aa4cd9)...
An error occurred while installing cryptography==3.3.1; python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5' --hash=sha256:84ef7a0c10c24a7773163f917f1cb6b4444597efd505a8aed0a22e8c4780f27e --hash=sha256:788a3c9942df5e4371c199d10383f44a105d67d401fb4304178020142f020244 --hash=sha256:9f6b0492d111b43de5f70052e24c1f0951cb9e6022188ebcb1cc3a3d301469b0 --hash=sha256:a69bd3c68b98298f490e84519b954335154917eaab52cf582fa2c5c7efc6e812 --hash=sha256:b4890d5fb9b7a23e3bf8abf5a8a7da8e228f1e97dc96b30b95685df840b6914a --hash=sha256:0003a52a123602e1acee177dc90dd201f9bb1e73f24a070db7d36c588e8f5c7d --hash=sha256:7e177e4bea2de937a584b13645cab32f25e3d96fc0bc4a4cf99c27dc77682be6 --hash=sha256:dc42f645f8f3a489c3dd416730a514e7a91a59510ddaadc09d04224c098d3302 --hash=sha256:69e836c9e5ff4373ce6d3ab311c1a2eed274793083858d3cd4c7d12ce20d5f9c --hash=sha256:9e21301f7a1e7c03dbea73e8602905a4ebba641547a462b26dd03451e5769e7c --hash=sha256:0e85aaae861d0485eb5a79d33226dd6248d2a9f133b81532c8f5aae37de10ff7 --hash=sha256:c366df0401d1ec4e548bebe8f91d55ebcc0ec3137900d214dd7aac8427ef3030 --hash=sha256:594a1db4511bc4d960571536abe21b4e5c3003e8750ab8365fafce71c5d86901 --hash=sha256:83d9d2dfec70364a74f4e7c70ad04d3ca2e6a08b703606993407bf46b97868c5! Will try again.
     ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 10/10 — 00:00:08
Installing initially failed dependencies...
[InstallError]:   File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/cli/command.py", line 233, in install
[InstallError]:       retcode = do_install(
[InstallError]:   File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 2052, in do_install
[InstallError]:       do_init(
[InstallError]:   File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 1304, in do_init
[InstallError]:       do_install_dependencies(
[InstallError]:   File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 899, in do_install_dependencies
[InstallError]:       batch_install(
[InstallError]:   File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 796, in batch_install
[InstallError]:       _cleanup_procs(procs, failed_deps_queue, retry=retry)
[InstallError]:   File "/usr/local/Cellar/pipenv/2020.11.15/libexec/lib/python3.9/site-packages/pipenv/core.py", line 703, in _cleanup_procs
[InstallError]:       raise exceptions.InstallError(c.dep.name, extra=err_lines)
[pipenv.exceptions.InstallError]: Looking in indexes: https://pypi.python.org/simple
[pipenv.exceptions.InstallError]: Collecting cryptography==3.3.1
[pipenv.exceptions.InstallError]:   Using cached cryptography-3.3.1.tar.gz (539 kB)
[pipenv.exceptions.InstallError]: ERROR: Disabling PEP 517 processing is invalid: project specifies a build backend of setuptools.build_meta in pyproject.toml
ERROR: Couldn't install package: cryptography
 Package installation failed...
  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/1 — 00:00:01

If instead of the apns2 line in the Pipfile I substitute cryptography explicitly (crytography = "*"), it fails too in the same way.

Dylan
  • 2,315
  • 2
  • 20
  • 33
  • `python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'` looks like wrong Python somehow (it should be `python_version not in ('3.0', '3.1', '3.2', '3.3', '3.4', '3.5')`, unless it's guaranteed that `python_version` includes the minor component of the version too), but I can't find it in the source code of `cryptography` package. Perhaps it's being used as a dependency of another package? – user202729 Jan 11 '21 at 01:49
  • That is, assuming that it's interpreted as Python (I'm not very family with Python packaging) -- but everyone seems to be doing that, so perhaps you can add a print statement there? – user202729 Jan 11 '21 at 01:51
  • 2
    Do you have the M1 chip in your MacBook? – hoefling Jan 11 '21 at 14:41
  • No, it's Intel-based. – Dylan Jan 11 '21 at 18:15

5 Answers5

12

If you are on a recent macOS version, the default pip version in your pipenv virtualenv incorrectly detects your OS version which makes looking up a compatible wheel fail (and it will start compiling cryptography dependencies).

To avoid the issue at it's root you can upgrade pip (in the virtualenv) to a version that detects mac os correctly:

pipenv shell
pip install -U pip

The bug was fixed in pip version 20.3 (2020-11-30) https://github.com/pypa/pip/issues/9138

Nikolai R Kristiansen
  • 535
  • 5
  • 11
11

The confusing error messages are caused by the fact that if/when pipenv fails to build cryptography for some reason, it tries a second time with --no-use-pep517 (see my comment below) and then we see those PEP 517 disabling errors instead of the root cause!

To track down your actual issue, try pip install cryptography (not pipenv) so you can see the root cause.

It is likely that you need to install rust and/or cargo, or openssl on your platform using apt-get install, apk add, brew install or whatever, then try again.

Once you get the issue resolved that way, pipenv installation should work.

See also cryptography package github issue #5773

DrMeers
  • 4,117
  • 2
  • 36
  • 38
  • 1
    See [these](https://github.com/pypa/pipenv/blob/47d8b96e10e462169cb9d3e719318565514408ad/pipenv/core.py#L772-L773) [lines](https://github.com/pypa/pipenv/blob/47d8b96e10e462169cb9d3e719318565514408ad/pipenv/core.py#L707-L713) in `pipenv` which I believe are the cause of the error masking – DrMeers Feb 09 '21 at 23:42
  • Also beware of cached artifacts try starting from scratch – elad silver Sep 15 '22 at 11:47
8

It seems the cryptography package can only be installed with pip, not pipenv, so by doing this first...

brew install pkg-config libffi openssl
env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install cryptography

(Commands taken from this question.)

...I could then run pipenv install which built and installed everything fine.

Dylan
  • 2,315
  • 2
  • 20
  • 33
  • See also [this Github issue](https://github.com/pyca/cryptography/issues/5773) which includes a few useful comments including a `pipenv` workaround – DrMeers Feb 09 '21 at 22:27
  • See also my answer below -- it *can* actually be installed via `pipenv`, installation errors are just hidden by `pipenv` due to an interesting quirk – DrMeers Mar 12 '21 at 22:29
2

I had a somewhat similar problem (with cffi instead of cryptography). I upgraded my pipenv version (specifically for me using pip3 install --upgrade pipenv) and it solved the problem.

Dean Gurvitz
  • 854
  • 1
  • 10
  • 24
1

The idea that cryptography isn't pipenv-installable can be debunked:

EDIT: I just saw on https://pypi.org/project/cryptography/#history that v3.4.3 was released just hours ago.

 mkdir cryptography_test
❯ cd cryptography_test
❯ pipenv --version
pipenv, version 2020.11.15
❯ pip --version
pip 20.3.1 from /home/reinv/.local/lib/python3.8/site-packages/pip (python 3.8)
❯ pipenv install cryptography
Creating a virtualenv for this project...
Pipfile: /home/reinv/Documents/cryptography_test/Pipfile
Using /usr/bin/python3.8 (3.8.5) to create virtualenv...
⠼ Creating virtual environment...created virtual environment CPython3.8.5.final.0-64 in 246ms
  creator CPython3Posix(dest=/home/reinv/.local/share/virtualenvs/cryptography_test-tHdlLBeP, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/reinv/.local/share/virtualenv)
    added seed packages: pip==20.3.1, setuptools==51.1.2, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /home/reinv/.local/share/virtualenvs/cryptography_test-tHdlLBeP
Creating a Pipfile for this project...
Installing cryptography...
Adding cryptography to Pipfile's [packages]...
✔ Installation Succeeded 
Pipfile.lock not found, creating...
Locking [dev-packages] dependencies...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (8ed4d5)!
Installing dependencies from Pipfile.lock (8ed4d5)...
     ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
❯ pipenv shell
Launching subshell in virtual environment...
❯  . /home/reinv/.local/share/virtualenvs/cryptography_test-tHdlLBeP/bin/activate
❯ pip list
Package      Version
------------ -------
cffi         1.14.4
cryptography 3.4.3
pip          20.3.1
pycparser    2.20
setuptools   51.1.2
wheel        0.36.2

Why this now does work, I have no idea and I'm not helping if I dump a complete set of installed ubuntu packages on my system. And yeah, cryptography suddenly started to break builds for ansible (as a sub-dependency) for me as well (which brought me here). I'm using pipenv too, but the question is why pipenv wouldn't be able to install the package in the first place, as @Dylan suggested, since as far as I can tell, pipenv uses pip as a package installer too.

I had to look up what the strange "Disabling PEP 517 processing is invalid" message meant, but I think it's explained here: https://github.com/pypa/pipenv/issues/4481#issuecomment-706376607

In short: first a compilation step fails, then a workaround is tried with an --no-use-pep517 option included somewhere that sends you a bit on a wild goose chase. (I haven't bothered reading the entire PEP 517 since it's as long as both my arms, but I fail to comprehend even the abstract.)

Rein
  • 78
  • 6