Virtualenv : Running `Scrapy` in a virtualenv env using python 3.5 giving error

Question

I am trying to run scrapy in a virtualenv environment with different python versions.

For python3, I did the following :

virtualenv -p python3 NAME
cd NAME    
source /bin/activate

After activation,

pip3 install scrapy

It is installed successfully. But when I run scrapy, I get the following error :

ImportError: /usr/local/lib/python2.7/dist-packages/cryptography/hazmat/bindings/_openssl.so: undefined symbol: PyInt_FromLong

Which shows that it is using my system's python2.7.

My ultimate goal is to run different versions of scrapy with different versions of python.

EDIT : I think there is a problem with cryptography module.

When I tried pip3 install --upgrade pyOpenSSL,

I got this :

Collecting pyOpenSSL
  Using cached pyOpenSSL-17.0.0-py2.py3-none-any.whl
Collecting cryptography>=1.7 (from pyOpenSSL)
  Using cached cryptography-1.8.1.tar.gz
Requirement already up-to-date: six>=1.5.2 in ./lib/python3.5/site-packages (from pyOpenSSL)
Requirement already up-to-date: idna>=2.1 in ./lib/python3.5/site-packages (from cryptography>=1.7->pyOpenSSL)
Requirement already up-to-date: asn1crypto>=0.21.0 in ./lib/python3.5/site-packages (from cryptography>=1.7->pyOpenSSL)
Requirement already up-to-date: packaging in /usr/local/lib/python2.7/dist-packages (from cryptography>=1.7->pyOpenSSL)
Collecting setuptools>=11.3 (from cryptography>=1.7->pyOpenSSL)
  Using cached setuptools-35.0.2-py2.py3-none-any.whl
Collecting cffi>=1.4.1 (from cryptography>=1.7->pyOpenSSL)
  Using cached cffi-1.10.0-cp35-cp35m-manylinux1_x86_64.whl
Requirement already up-to-date: pyparsing in /usr/local/lib/python2.7/dist-packages (from packaging->cryptography>=1.7->pyOpenSSL)
Requirement already up-to-date: appdirs>=1.4.0 in /usr/local/lib/python2.7/dist-packages (from setuptools>=11.3->cryptography>=1.7->pyOpenSSL)
Requirement already up-to-date: pycparser in /usr/local/lib/python2.7/dist-packages (from cffi>=1.4.1->cryptography>=1.7->pyOpenSSL)
Building wheels for collected packages: cryptography
  Running setup.py bdist_wheel for cryptography ... error
  Complete output from command /home/parth/gsoc/sample/try/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-nvvy22iz/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp66xa7d7xpip-wheel- --python-tag cp35:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-build-nvvy22iz/cryptography/setup.py", line 335, in <module>
      **keywords_with_side_effects(sys.argv)
    File "/usr/lib/python3.5/distutils/core.py", line 108, in setup
      _setup_distribution = dist = klass(attrs)
    File "/usr/local/lib/python2.7/dist-packages/setuptools/dist.py", line 320, in __init__
      _Distribution.__init__(self, attrs)
    File "/usr/lib/python3.5/distutils/dist.py", line 281, in __init__
      self.finalize_options()
    File "/usr/local/lib/python2.7/dist-packages/setuptools/dist.py", line 387, in finalize_options
      ep.load()(self, ep.name, value)
    File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 187, in cffi_modules
      add_cffi_module(dist, cffi_module)
    File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
      execfile(build_file_name, mod_vars)
    File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 25, in execfile
      exec(code, glob, glob)
    File "src/_cffi_src/build_openssl.py", line 94, in <module>
      extra_link_args=extra_link_args(compiler_type()),
    File "/tmp/pip-build-nvvy22iz/cryptography/src/_cffi_src/utils.py", line 61, in build_ffi_for_binding
      extra_link_args=extra_link_args,
    File "/tmp/pip-build-nvvy22iz/cryptography/src/_cffi_src/utils.py", line 69, in build_ffi
      ffi = FFI()
    File "/usr/local/lib/python2.7/dist-packages/cffi/api.py", line 56, in __init__
      import _cffi_backend as backend
  ImportError: /usr/local/lib/python2.7/dist-packages/_cffi_backend.so: undefined symbol: _Py_ZeroStruct

  ----------------------------------------
  Failed building wheel for cryptography

Running setup.py clean for cryptography
  Complete output from command /home/parth/gsoc/sample/try/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-nvvy22iz/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" clean --all:
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/tmp/pip-build-nvvy22iz/cryptography/setup.py", line 335, in <module>
      **keywords_with_side_effects(sys.argv)
    File "/usr/lib/python3.5/distutils/core.py", line 108, in setup
      _setup_distribution = dist = klass(attrs)
    File "/usr/local/lib/python2.7/dist-packages/setuptools/dist.py", line 320, in __init__
      _Distribution.__init__(self, attrs)
    File "/usr/lib/python3.5/distutils/dist.py", line 281, in __init__
      self.finalize_options()
    File "/usr/local/lib/python2.7/dist-packages/setuptools/dist.py", line 387, in finalize_options
      ep.load()(self, ep.name, value)
    File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 187, in cffi_modules
      add_cffi_module(dist, cffi_module)
    File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
      execfile(build_file_name, mod_vars)
    File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 25, in execfile
      exec(code, glob, glob)
    File "src/_cffi_src/build_openssl.py", line 94, in <module>
      extra_link_args=extra_link_args(compiler_type()),
    File "/tmp/pip-build-nvvy22iz/cryptography/src/_cffi_src/utils.py", line 61, in build_ffi_for_binding
      extra_link_args=extra_link_args,
    File "/tmp/pip-build-nvvy22iz/cryptography/src/_cffi_src/utils.py", line 69, in build_ffi
      ffi = FFI()
    File "/usr/local/lib/python2.7/dist-packages/cffi/api.py", line 56, in __init__
      import _cffi_backend as backend
  ImportError: /usr/local/lib/python2.7/dist-packages/_cffi_backend.so: undefined symbol: _Py_ZeroStruct

  ----------------------------------------
  Failed cleaning build dir for cryptography
Failed to build cryptography
Installing collected packages: setuptools, cffi, cryptography, pyOpenSSL
  Found existing installation: setuptools 34.3.2
    Not uninstalling setuptools at /usr/local/lib/python2.7/dist-packages, outside environment /home/parth/gsoc/sample/try
  Found existing installation: cffi 1.9.1
    Not uninstalling cffi at /usr/local/lib/python2.7/dist-packages, outside environment /home/parth/gsoc/sample/try
  Found existing installation: cryptography 1.5.3
    Not uninstalling cryptography at /usr/local/lib/python2.7/dist-packages, outside environment /home/parth/gsoc/sample/try
  Running setup.py install for cryptography ... error
    Complete output from command /home/parth/gsoc/sample/try/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-nvvy22iz/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-tdsypb26-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/parth/gsoc/sample/try/include/site/python3.5/cryptography:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-nvvy22iz/cryptography/setup.py", line 335, in <module>
        **keywords_with_side_effects(sys.argv)
      File "/usr/lib/python3.5/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/usr/local/lib/python2.7/dist-packages/setuptools/dist.py", line 320, in __init__
        _Distribution.__init__(self, attrs)
      File "/usr/lib/python3.5/distutils/dist.py", line 281, in __init__
        self.finalize_options()
      File "/usr/local/lib/python2.7/dist-packages/setuptools/dist.py", line 387, in finalize_options
        ep.load()(self, ep.name, value)
      File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 187, in cffi_modules
        add_cffi_module(dist, cffi_module)
      File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 49, in add_cffi_module
        execfile(build_file_name, mod_vars)
      File "/usr/local/lib/python2.7/dist-packages/cffi/setuptools_ext.py", line 25, in execfile
        exec(code, glob, glob)
      File "src/_cffi_src/build_openssl.py", line 94, in <module>
        extra_link_args=extra_link_args(compiler_type()),
      File "/tmp/pip-build-nvvy22iz/cryptography/src/_cffi_src/utils.py", line 61, in build_ffi_for_binding
        extra_link_args=extra_link_args,
      File "/tmp/pip-build-nvvy22iz/cryptography/src/_cffi_src/utils.py", line 69, in build_ffi
        ffi = FFI()
      File "/usr/local/lib/python2.7/dist-packages/cffi/api.py", line 56, in __init__
        import _cffi_backend as backend
    ImportError: /usr/local/lib/python2.7/dist-packages/_cffi_backend.so: undefined symbol: _Py_ZeroStruct

    ----------------------------------------
  Can't rollback cryptography, nothing uninstalled.

@RemyJ It was activated. I did `cd` to enter it. I should write that too. — Parth Verma, May 19 '17 at 18:06
@Matt.St Same error. I guess there is some problem with my system. — Parth Verma, May 19 '17 at 18:10
try looking at http://stackoverflow.com/questions/35125752/error-installing-scrapy-in-python-3-5 — Matt. Stroh, May 19 '17 at 18:11
There might be an issue with `cryptography` module. When I tried `pip3 install --upgrade pyOpenSSL`, I got this : `Failed building wheel for cryptography Failed cleaning build dir for cryptography Can't rollback cryptography, nothing uninstalled.` — Parth Verma, May 19 '17 at 18:23

score -1 · Answer 1 · answered May 19 '17 at 18:32

-1

Try with

pip install scrapy

or

sudo pip install scrapy

And when you write python3 which version open?

answered May 19 '17 at 18:32

IsaBostan

60
4

Why sudo? Also question says it is installed fine – OneCricketeer May 19 '17 at 18:34
I thought it was installing problem. What about python part – IsaBostan May 19 '17 at 18:37
It's a virtualenv, so what about it? – OneCricketeer May 19 '17 at 19:06
When you run python3, which python runs, python2 or python3.5 – IsaBostan May 19 '17 at 22:07
That's a question for the OP, not me. But again, it's a virtualenv, so there should only be one version available – OneCricketeer May 20 '17 at 06:39
I am sayin that: goto command line and write, python3 press enter. You will see python version. Which python is that? That might be version problem, if it is uninstall python3 then install again. – IsaBostan May 20 '17 at 15:30

Virtualenv : Running `Scrapy` in a virtualenv env using python 3.5 giving error

1 Answers1