11

When I try to install Scrapy on ubuntu 13.04 with the command sudo easy_install -U Scrapy, I get the following error:

Searching for Scrapy
Reading pypi.python.org/simple/Scrapy/
Best match: Scrapy 0.16.5
Processing Scrapy-0.16.5-py2.7.egg
Scrapy 0.16.5 is already the active version in easy-install.pth
Installing scrapy script to /usr/local/bin

Using /usr/local/lib/python2.7/dist-packages/Scrapy-0.16.5-py2.7.egg
Processing dependencies for Scrapy
Searching for Twisted>=8.0
Reading pypi.python.org/simple/Twisted/
Best match: Twisted 13.1.0
Downloading pypi.python.org/packages/source/T/Twisted/Twisted-13.1.0.tar.bz2#md5=5609c91ed465f5a7da48d30a0e7b6960
Processing Twisted-13.1.0.tar.bz2
Writing /tmp/easy_install-xYafJF/Twisted-13.1.0/setup.cfg
Running Twisted-13.1.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-xYafJF/Twisted-13.1.0/egg-dist-tmp-ybetxt
twisted/runner/portmap.c:10:20: fatal error: Python.h: No such file or directory
compilation terminated.
error: Setup script exited with error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

I also tried pip install Scrapy command but again I get the same error x86_64-linux-gnu-gcc

Could you please help me to install Scrapy on Ubuntu 13.04?

dustin
  • 4,309
  • 12
  • 57
  • 79
Burak Karacay
  • 113
  • 1
  • 4
  • Yes I got the "gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1) " – Burak Karacay Jul 12 '13 at 09:13
  • Official scrapy web site does not recomment this. "Don’t use the python-scrapy package provided by Ubuntu, they are typically too old and slow to catch up with latest Scrapy." – Burak Karacay Jul 12 '13 at 09:15
  • Yes. It gives the same error. "error: command 'x86_64-linux-gnu-gcc' failed with exit status 1" "Command /usr/bin/python -c "import setuptools;__file__='/tmp/pip-build-brk/Twisted/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-aiQgfx-record/install-record.txt --single-version-externally-managed failed with error code 1 in /tmp/pip-build-brk/Twisted" – Burak Karacay Jul 12 '13 at 09:17
  • 1
    Do you have python dev installed? – dustin Jul 12 '13 at 09:19
  • Thanks a lot. After installing "python2.7-dev", I succeded to install Scrapy – Burak Karacay Jul 12 '13 at 09:26

4 Answers4

17

For me, I had python dev installed but still wasnt working (was failing when trying to install cryptography) and I found this stack overflow post which says to run this command to guarantee all dependencies are installed

sudo apt-get install build-essential libssl-dev libffi-dev python-dev libxml2-dev

and that worked for me

Community
  • 1
  • 1
Bobby
  • 6,840
  • 1
  • 22
  • 25
  • After installing these packages, it worked for me: libssl-doc, libssl-dev, libxml2-dev. I'm still not certain if any of them are superfluous. – emmagras Jan 06 '15 at 17:39
12

In order to install scrapy, you need to have python dev installed first.

dustin
  • 4,309
  • 12
  • 57
  • 79
  • 3
    This should be listed as a requirement: https://pypi.python.org/pypi/Scrapy if it is a requirement. For me, it certainly was. – Alex A. Sep 21 '13 at 07:21
  • 1
    In my case "libxslt1-dev" was also missing – KevinS Dec 27 '13 at 12:47
  • This isn't helping me. I have the following error: python-dev : Depends: libpython-dev (= 2.7.4-0ubuntu1) but it is not going to be installed Depends: python2.7-dev (>= 2.7.4-1~) but it is not going to be installed N: Ignoring file 'canonical' in directory '/etc/apt/sources.list.d/' as it has no filename extension N: Ignoring file 'canonical' in directory '/etc/apt/sources.list.d/' as it has no filename extension E: Unable to correct problems, you have held broken packages – Lawrence DeSouza Mar 23 '14 at 15:39
3

Use this : sudo apt-get install libssl-dev

before installing scrapy. and then

sudo pip install scrapy --force --upgrade

Manoj Sahu
  • 2,774
  • 20
  • 18
1

I had the exact same issue occuring when trying to install Scrapy on a DigitalOcean droplet, it wasn't a library issue, apparently it can happen in low RAM box, you can solve it by combining ionice + pip:

sudo ionice -c3 pip install Scrapy
wtrevino
  • 4,831
  • 2
  • 15
  • 9