Scrapy setup ubuntu 16.04 or any other

Question

My today installed fresh Ubuntu 16.04 was preinstalled with python:

p@Scrapy:~$ python --version
Python 2.7.11+
p@Scrapy:~$ python3 --version
Python 3.5.1+

As mentioned on manual page http://doc.scrapy.org/en/latest/intro/install.html I open this link http://doc.scrapy.org/en/latest/topics/ubuntu.html#topics-ubuntu and tried to follow steps described to install Scrapy.

But I get an error after step 3

sudo apt-get update && sudo apt-get install scrapy

...

The following packages have unmet dependencies:
 scrapy : Depends: python-support (>= 0.90.0) but it is not installable
          Recommends: python-setuptools but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

I post a problem yesterday about error with Scrapy and looks like it was setup issue too but on Windows. Cannot setup Scrapy on windows So I tried today with Ubuntu and once more without luck.

So question how to setup Scrapy on Ubuntu 16.04 or may be other version? Looks like Scrapy manuals outdated. I would think Scrapy project died, but I know that people still use Scrapy. May be Scrapy only works with python 2.+? So I will stay with Windows. Cannot check all variants. It takes too much time. Can anyone mention stable config (operating system + python version) working with Scrapy?

Thanks.

Update

Here I tried with Docker. I create Dockerfile, other steps from terminal:

p@ScrapyPython3:~$ cat Dockerfile
$ cat Dockerfile
FROM ubuntu:xenial

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update

# Install Python3 and dev headers
RUN apt-get install -y \
    python3 \
    python-dev \
    python3-dev

# Install cryptography
RUN apt-get install -y \
    build-essential \
    libssl-dev \
    libffi-dev

# install lxml
RUN apt-get install -y \
    libxml2-dev \
    libxslt-dev

# install pip
RUN apt-get install -y python-pip

RUN useradd --create-home --shell /bin/bash scrapyuser

USER scrapyuser
WORKDIR /home/scrapyuser
p@ScrapyPython3:~$ sudo docker build -t redapple/scrapy-ubuntu-xenial .
Sending build context to Docker daemon 81.21 MB
Step 1 : $ 
Unknown instruction: $
p@ScrapyPython3:~$ sudo docker run -t -i redapple/scrapy-ubuntu-xenial
Unable to find image 'redapple/scrapy-ubuntu-xenial:latest' locally
Pulling repository docker.io/redapple/scrapy-ubuntu-xenial
docker: Error: image redapple/scrapy-ubuntu-xenial not found.
See 'docker run --help'.
p@ScrapyPython3:~$ pip install scrapy
Requirement already satisfied (use --upgrade to upgrade): scrapy in ./.local/lib/python2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): queuelib in ./.local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in ./.local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in ./.local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in ./.local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.14.2 in ./.local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): service-identity in ./.local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in ./.local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in ./.local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): parsel>=0.9.3 in ./.local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): PyDispatcher>=2.0.5 in ./.local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): cryptography>=1.3 in ./.local/lib/python2.7/site-packages (from pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in ./.local/lib/python2.7/site-packages (from Twisted>=10.0.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules in ./.local/lib/python2.7/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in ./.local/lib/python2.7/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): attrs in ./.local/lib/python2.7/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools>=11.3 in ./.local/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): ipaddress in ./.local/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): enum34 in ./.local/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): idna>=2.0 in ./.local/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): cffi>=1.4.1 in ./.local/lib/python2.7/site-packages (from cryptography>=1.3->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pycparser in ./.local/lib/python2.7/site-packages (from cffi>=1.4.1->cryptography>=1.3->pyOpenSSL->scrapy)
p@ScrapyPython3:~$ scrapy version
The program 'scrapy' is currently not installed. You can install it by typing:
sudo apt install python-scrapy

UPDATE 1 Looks like first row in Dockerfile should not be there. If i remove it ($ cat Dockerfile)

I can get docker image up and running, but once more no luck with pip install scrapy. More over I realize I have to do some overhead with docker to be able upload scrapy files there (I prefer GUI when coding). It there any tool installed in this Docker image? Here install log:

p@ScrapyPython3:~$ sudo docker run -t -i redapple/scrapy-ubuntu-xenial
    scrapyuser@41bef38de45d:~$ python --version   
    Python 2.7.11+
    scrapyuser@41bef38de45d:~$ python3 --version
    Python 3.5.1+
    scrapyuser@41bef38de45d:~$ pip install scrapy
    Collecting scrapy
      Downloading Scrapy-1.1.0-py2.py3-none-any.whl (294kB)
        100% |################################| 296kB 245kB/s 
    Collecting queuelib (from scrapy)
      Downloading queuelib-1.4.2-py2.py3-none-any.whl
    Collecting pyOpenSSL (from scrapy)
      Downloading pyOpenSSL-16.0.0-py2.py3-none-any.whl (45kB)
        100% |################################| 51kB 12.7MB/s 
    Collecting Twisted>=10.0.0 (from scrapy)
      Downloading Twisted-16.2.0.tar.bz2 (2.9MB)
        100% |################################| 2.9MB 472kB/s 
    Collecting six>=1.5.2 (from scrapy)
      Downloading six-1.10.0-py2.py3-none-any.whl
    Collecting w3lib>=1.14.2 (from scrapy)
      Downloading w3lib-1.14.2-py2.py3-none-any.whl
    Collecting service-identity (from scrapy)
      Downloading service_identity-16.0.0-py2.py3-none-any.whl
    Collecting cssselect>=0.9 (from scrapy)
      Downloading cssselect-0.9.2-py2.py3-none-any.whl
    Collecting lxml (from scrapy)
      Downloading lxml-3.6.0.tar.gz (3.7MB)
        100% |################################| 3.7MB 389kB/s 
    Collecting parsel>=0.9.3 (from scrapy)
      Downloading parsel-1.0.2-py2.py3-none-any.whl
    Collecting PyDispatcher>=2.0.5 (from scrapy)
      Downloading PyDispatcher-2.0.5.tar.gz
    Collecting cryptography>=1.3 (from pyOpenSSL->scrapy)
      Downloading cryptography-1.4.tar.gz (399kB)
        100% |################################| 409kB 1.4MB/s 
    Collecting zope.interface>=3.6.0 (from Twisted>=10.0.0->scrapy)
      Downloading zope.interface-4.2.0.tar.gz (146kB)
        100% |################################| 153kB 1.2MB/s 
    Collecting pyasn1-modules (from service-identity->scrapy)
      Downloading pyasn1_modules-0.0.8-py2.py3-none-any.whl
    Collecting pyasn1 (from service-identity->scrapy)
      Downloading pyasn1-0.1.9-py2.py3-none-any.whl
    Collecting attrs (from service-identity->scrapy)
      Downloading attrs-16.0.0-py2.py3-none-any.whl
    Collecting idna>=2.0 (from cryptography>=1.3->pyOpenSSL->scrapy)
      Downloading idna-2.1-py2.py3-none-any.whl (54kB)
        100% |################################| 61kB 10.8MB/s 
    Collecting setuptools>=11.3 (from cryptography>=1.3->pyOpenSSL->scrapy)
      Downloading setuptools-23.0.0-py2.py3-none-any.whl (435kB)
        100% |################################| 440kB 1.2MB/s 
    Collecting enum34 (from cryptography>=1.3->pyOpenSSL->scrapy)
      Downloading enum34-1.1.6-py2-none-any.whl
    Collecting ipaddress (from cryptography>=1.3->pyOpenSSL->scrapy)
      Downloading ipaddress-1.0.16-py27-none-any.whl
    Collecting cffi>=1.4.1 (from cryptography>=1.3->pyOpenSSL->scrapy)
      Downloading cffi-1.6.0.tar.gz (397kB)
        100% |################################| 399kB 1.3MB/s 
    Collecting pycparser (from cffi>=1.4.1->cryptography>=1.3->pyOpenSSL->scrapy)
      Downloading pycparser-2.14.tar.gz (223kB)
        100% |################################| 225kB 1.1MB/s 
    Building wheels for collected packages: Twisted, lxml, PyDispatcher, cryptography, zope.interface, cffi, pycparser
      Running setup.py bdist_wheel for Twisted ... done
      Stored in directory: /home/scrapyuser/.cache/pip/wheels/fe/9d/3f/9f7b1c768889796c01929abb7cdfa2a9cdd32bae64eb7aa239
      Running setup.py bdist_wheel for lxml ... done
      Stored in directory: /home/scrapyuser/.cache/pip/wheels/6c/eb/a1/e4ff54c99630e3cc6ec659287c4fd88345cd78199923544412
      Running setup.py bdist_wheel for PyDispatcher ... done
      Stored in directory: /home/scrapyuser/.cache/pip/wheels/86/02/a1/5857c77600a28813aaf0f66d4e4568f50c9f133277a4122411
      Running setup.py bdist_wheel for cryptography ... done
      Stored in directory: /home/scrapyuser/.cache/pip/wheels/f6/6c/21/11ec069285a52d7fa8c735be5fc2edfb8b24012c0f78f93d20
      Running setup.py bdist_wheel for zope.interface ... done
      Stored in directory: /home/scrapyuser/.cache/pip/wheels/20/a2/bc/74fe87cee17134f5219ba01fe82dd8c10998377e0fb910bb22
      Running setup.py bdist_wheel for cffi ... done
      Stored in directory: /home/scrapyuser/.cache/pip/wheels/8f/00/29/553c1b1db38bbeec3fec428ae4e400cd8349ecd99fe86edea1
      Running setup.py bdist_wheel for pycparser ... done
      Stored in directory: /home/scrapyuser/.cache/pip/wheels/9b/f4/2e/d03e949a551719a1ffcb659f2c63d8444f4df12e994ce52112
    Successfully built Twisted lxml PyDispatcher cryptography zope.interface cffi pycparser
    Installing collected packages: queuelib, idna, pyasn1, six, setuptools, enum34, ipaddress, pycparser, cffi, cryptography, pyOpenSSL, zope.interface, Twisted, w3lib, pyasn1-modules, attrs, service-identity, cssselect, lxml, parsel, PyDispatcher, scrapy
    Successfully installed PyDispatcher Twisted attrs cffi cryptography cssselect enum34 idna ipaddress lxml parsel pyOpenSSL pyasn1 pyasn1-modules pycparser queuelib scrapy service-identity setuptools-20.7.0 six w3lib zope.interface
    You are using pip version 8.1.1, however version 8.1.2 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.
    scrapyuser@41bef38de45d:~$ scrapy version
    bash: scrapy: command not found

score 2 · Answer 1 · edited May 23 '17 at 11:58

The Scrapy install docs need updating. Really sorry about that.

Ubuntu packages from http://archive.scrapy.org/ubuntu are not up to date (as I write these lines on 2016-06-15) so do not use them if you want latest (Py3 compatible) scrapy

The page you linked, http://doc.scrapy.org/en/latest/intro/install.html#ubuntu-9-10-or-above, has an alternative setup using pip with (quite a few) dependencies:

If you prefer to build the python dependencies locally instead of relying on system packages you’ll need to install their required non-python dependencies first:

    sudo apt-get install python-dev python-pip libxml2-dev libxslt1-dev zlib1g-dev libffi-dev libssl-dev

You can install Scrapy with pip after that:

    pip install Scrapy

Also check https://stackoverflow.com/a/37677910/2572383

I would recommend installing all these below, if you want Python 2 and Python 3:

apt-get install -y \
    python3 \
    python-dev \
    python3-dev

# for cryptography
apt-get install -y \
    build-essential \
    libssl-dev \
    libffi-dev

# for lxml
apt-get install -y \
    libxml2-dev \
    libxslt-dev

# install pip (if not already installed)
apt-get install -y python-pip

Another recommendation: install virtualenvwrapper so you can create a local Python 3 virtual environment:

$ mkvirtualenv --python=/usr/bin/python3 scrapy.py3
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/paul/.virtualenvs/scrapy.py3/bin/python3
Also creating executable in /home/paul/.virtualenvs/scrapy.py3/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
virtualenvwrapper.user_scripts creating /home/paul/.virtualenvs/scrapy.py3/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/paul/.virtualenvs/scrapy.py3/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/paul/.virtualenvs/scrapy.py3/bin/preactivate
virtualenvwrapper.user_scripts creating /home/paul/.virtualenvs/scrapy.py3/bin/postactivate
virtualenvwrapper.user_scripts creating /home/paul/.virtualenvs/scrapy.py3/bin/get_env_details

And then simply pip install scrapy inside the virtual environment:

(scrapy.py3) paul@paul-SATELLITE-R830:~/src/scrapy.org$ pip install --upgrade --no-cache-dir scrapy
Collecting scrapy
  Downloading Scrapy-1.1.0-py2.py3-none-any.whl (294kB)
    100% |████████████████████████████████| 296kB 1.7MB/s 
Collecting cssselect>=0.9 (from scrapy)
  Downloading cssselect-0.9.1.tar.gz
Collecting queuelib (from scrapy)
  Downloading queuelib-1.4.2-py2.py3-none-any.whl
Collecting parsel>=0.9.3 (from scrapy)
  Downloading parsel-1.0.2-py2.py3-none-any.whl
Collecting Twisted>=10.0.0 (from scrapy)
  Downloading Twisted-16.2.0.tar.bz2 (2.9MB)
    100% |████████████████████████████████| 2.9MB 1.9MB/s 
Collecting lxml (from scrapy)
  Downloading lxml-3.6.0.tar.gz (3.7MB)
    100% |████████████████████████████████| 3.7MB 2.0MB/s 
Collecting PyDispatcher>=2.0.5 (from scrapy)
  Downloading PyDispatcher-2.0.5.tar.gz
Collecting six>=1.5.2 (from scrapy)
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting pyOpenSSL (from scrapy)
  Downloading pyOpenSSL-16.0.0-py2.py3-none-any.whl (45kB)
    100% |████████████████████████████████| 51kB 2.1MB/s 
Collecting service-identity (from scrapy)
  Downloading service_identity-16.0.0-py2.py3-none-any.whl
Collecting w3lib>=1.14.2 (from scrapy)
  Downloading w3lib-1.14.2-py2.py3-none-any.whl
Collecting zope.interface>=4.0.2 (from Twisted>=10.0.0->scrapy)
  Downloading zope.interface-4.2.0.tar.gz (146kB)
    100% |████████████████████████████████| 153kB 2.1MB/s 
Collecting cryptography>=1.3 (from pyOpenSSL->scrapy)
  Downloading cryptography-1.4.tar.gz (399kB)
    100% |████████████████████████████████| 409kB 2.0MB/s 
Collecting attrs (from service-identity->scrapy)
  Downloading attrs-16.0.0-py2.py3-none-any.whl
Collecting pyasn1-modules (from service-identity->scrapy)
  Downloading pyasn1_modules-0.0.8-py2.py3-none-any.whl
Collecting pyasn1 (from service-identity->scrapy)
  Downloading pyasn1-0.1.9-py2.py3-none-any.whl
Requirement already up-to-date: setuptools in /home/paul/.virtualenvs/scrapy.py3/lib/python3.5/site-packages (from zope.interface>=4.0.2->Twisted>=10.0.0->scrapy)
Collecting idna>=2.0 (from cryptography>=1.3->pyOpenSSL->scrapy)
  Downloading idna-2.1-py2.py3-none-any.whl (54kB)
    100% |████████████████████████████████| 61kB 3.1MB/s 
Collecting cffi>=1.4.1 (from cryptography>=1.3->pyOpenSSL->scrapy)
  Downloading cffi-1.6.0.tar.gz (397kB)
    100% |████████████████████████████████| 399kB 2.1MB/s 
Collecting pycparser (from cffi>=1.4.1->cryptography>=1.3->pyOpenSSL->scrapy)
  Downloading pycparser-2.14.tar.gz (223kB)
    100% |████████████████████████████████| 225kB 1.9MB/s 
Installing collected packages: cssselect, queuelib, six, w3lib, lxml, parsel, zope.interface, Twisted, PyDispatcher, idna, pyasn1, pycparser, cffi, cryptography, pyOpenSSL, attrs, pyasn1-modules, service-identity, scrapy
  Running setup.py install for cssselect ... done
  Running setup.py install for lxml ... done
  Running setup.py install for zope.interface ... done
  Running setup.py install for Twisted ... done
  Running setup.py install for PyDispatcher ... done
  Running setup.py install for pycparser ... done
  Running setup.py install for cffi ... done
  Running setup.py install for cryptography ... done
Successfully installed PyDispatcher-2.0.5 Twisted-16.2.0 attrs-16.0.0 cffi-1.6.0 cryptography-1.4 cssselect-0.9.1 idna-2.1 lxml-3.6.0 parsel-1.0.2 pyOpenSSL-16.0.0 pyasn1-0.1.9 pyasn1-modules-0.0.8 pycparser-2.14 queuelib-1.4.2 scrapy-1.1.0 service-identity-16.0.0 six-1.10.0 w3lib-1.14.2 zope.interface-4.2.0

Hi. I just played with error in my post and find solution, though it bad. First I download missed package with: wget http://launchpadlibrarian.net/109052632/python-support_1.0.15_all.deb . And install it with: sudo dpkg -i python-support_1.0.15_all.deb . After that I repeat step 3 and Scrapy installed without errors. But Scrapy version 1.0.5.post4+g4b324a8, instead of 1.1. I will try your approach, but not get is it one solution or there are few different? Will try from first: alternative setup using pip. — P_M, Jun 15 '16 at 12:47
I tried first approach and it run without errors, but scrapy not works: p@Scrapy:~$ scrapy version bash: /usr/bin/scrapy: No such file or directory — P_M, Jun 15 '16 at 13:09
Well, looks like installation process highly dependent on operation system version and python version. I tried ubuntu 14.04 with python 2.7.6 preinstalled. I find opinion that default python on ubuntu should not be removed and left it as is. After that I take this manual http://linoxide.com/ubuntu-how-to/scrapy-install-ubuntu/ and it installed scrapy 1.1.0 within 5 minutes. Now I see that manuals about installation often miss valuable info about OS version, but highly rely on that. Now what route should I take to make scrapy work with python 3.3+? — P_M, Jun 16 '16 at 08:12
By the way I tried virtualenv but no luck either. Many of manuals are breaks on some steps where the packets versions differ. May be easier to run scrapy from already configured Virtualbox image or something like Docker? Also have to say many existing installation options make things difficult to check (multiply options number by 3 Ubuntu version and few python configs). It would be nice to see previously tested manual for installation process. — P_M, Jun 16 '16 at 08:16
@Pavlo, Scrapy itself is pure-python but depends on packages that are not easy to install (e.g. lxml). Scrapy install docs try to cover the steps to get you covered, but like you said, it can depend on what OS you're using, what packages you already have installed etc. I'm surprised that it failed with virtualenv. If you have time, please report installation issue with complete console logs at https://github.com/scrapy/scrapy/issues/new. http://stackoverflow.com/a/37677910/2572383 shows a complete Dockerfile to test on Ubuntu 16.04, so scrapy installs on a clean Docker image of Ubuntu 16.04. — paul trmbrth, Jun 16 '16 at 08:53
I install Docker, ant tried with it. No luck. Though here I not sure did everything ight, because not experienced with Docker. I update my post with Docker setup steps — P_M, Jun 17 '16 at 08:10

score 0 · Answer 2 · answered Jul 09 '17 at 04:27

I'm using Ubuntu 16.04 too and the following did work for me:

Download and install Anaconda, a data-centric Python distribution provided at no cost by Continuum Analytics. Currently (2017-07-09) Python 3.6 and Python 2.7 are available.
Decide which version of scrapy is needed in your project and use conda, their paket manager to figure out which versions are available in a default installation of Anaconda:
conda search scrapy yields

1.3.3 py36_0 defaults
If you can live with version 1.3.3, simply install it: conda install scrapy
According to the Scrapy website, the most recent stable version is 1.4.0 and you might want to go with that instead. You could probably run pip install scrapy to get it, but there is another way which I'd recommend:
Check the Anaconda cloud and realize that version 1.4.0 is available via conda-forge, a community-driven project in which numerous packages have been 'forged' to make them installable with conda.
You have now two options to install the most recent version from the conda-forge channel
- explicitly state the channel: conda install -c conda-forge scrapy=1.4.0 or
- add the conda-forge channel as the primary package source of your installation with conda config --add channels conda-forge and simply install scrapy with conda install scrapy then.

Thanks for posting this. Though I cannot test, because since last year not use Scrapy, because of setup issues. I just not sure I can use it in production (often I cannot choose server OS and it makes me impossible to predict deployment time). Of course it could still work with Docker or virtual machine, but for now I just skip Scrapy. Sorry for that because when I tried to read about Scrapy development, I found few great concepts and I use them now, but with different platform. — P_M, Jul 10 '17 at 07:10

Scrapy setup ubuntu 16.04 or any other

2 Answers2