4

I tried to install Scrapy on El Capitan but have not been successful yet. This happens when I use pip install Scrapy:

#include <openssl/opensslv.h>

         ^

1 error generated.

error: command 'cc' failed with exit status 1

----------------------------------------
Cleaning up...
Command /<scrapy_project>/venv/bin/python -c "import setuptools, tokenize;__file__='/<scrapy_project>/venv/build/cryptography/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/p6/jvf54l7d5c7dntzm6d3rfc3w0000gn/T/pip-D2QIZq-record/install-record.txt --single-version-externally-managed --compile --install-headers /<scrapy_project>/venv/include/site/python2.7 failed with error code 1 in /<scrapy_project>/venv/build/cryptography

My Xcode and Xcode command tools are up to date. I tried installing Scrapy with and without the approach via homebrew according to http://doc.scrapy.org/en/latest/intro/install.html#intro-install

Edit: I did the following:

  • brew install openssl && brew link openssl --force according to Craicerjack's suggestion
  • pip install cryptography
  • pip install scrapy

all worked without any errors. But scrapy --version throws this error:

ImportError: dlopen(/<scrapy_project>/venv/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so, 2): Symbol not found: _BIO_new_CMS
  Referenced from: /<scrapy_project>/venv/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so
  Expected in: flat namespace
 in /<scrapy_project>/venv/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so
tinaheidinger
  • 854
  • 1
  • 8
  • 21
  • 1
    looking at that looks like a problem with building the cryptography package. maybe `pip install cryptography` first? also [the second comment on this question](http://stackoverflow.com/questions/36215447/error-occurs-when-installing-cryptography-for-scrapy-in-virtualenv-on-os-x) might be of help – Craicerjack May 24 '16 at 11:53
  • You are missing a dev package for openssl – Padraic Cunningham May 24 '16 at 11:53
  • @Craicerjack thanks for your comment! please see the edit of my question. – tinaheidinger May 24 '16 at 12:14
  • 1
    well this question, http://stackoverflow.com/questions/33462779/symbol-not-found-bio-new-cms, seems to have the solution to that. – Craicerjack May 24 '16 at 12:27
  • 1
    Yay! You saved my day. Thank you @Craicerjack! You can post a solution that i would accept if you want. – tinaheidinger May 24 '16 at 12:52

1 Answers1

5

The issue here is with installation of the dependencies needed for Scrapy

first of all you should upgrade to the latest version of pip:

pip install --upgrade pip

If that doesnt work, to build cryptography and dynamically link it:

brew install openssl
env LDFLAGS="-L$(brew --prefix openssl)/lib" CFLAGS="-I$(brew --prefix openssl)/include" pip install cryptography

further information on installation can be found in the cryptography docs

Other issues may be solved by using the command 

LDFLAGS="-L/usr/local/opt/openssl/lib" pip install cryptography --no-use-wheel

This shouldnt be necessary however, if all your software (latest pip and cryptography), is up to date. More on this issue can be found in the issues in the cryptography repo on github

Craicerjack
  • 6,203
  • 2
  • 31
  • 39