2

I try to install Scrapy using easy install. I installed Cython and it seems to work, I installed libxml2 form http://www.lfd.uci.edu/~gohlke/pythonlibs/ I changed cygwinccompiler following this post scrapy installation error I still got an error and don't know what to do next. Also I don't know why there is that it is Building without Cython I have Windows 7 32-bit and Python 2.7 

C:\Users\t>easy_install -U Scrapy
Searching for Scrapy
Reading http://pypi.python.org/simple/Scrapy/
Reading http://scrapy.org
Best match: scrapy 0.16.1
Processing scrapy-0.16.1-py2.7.egg
scrapy 0.16.1 is already the active version in easy-install.pth
Installing scrapy script to C:\Python27\Scripts
Installing scrapy.bat script to C:\Python27\Scripts

Using c:\python27\lib\site-packages\scrapy-0.16.1-py2.7.egg
Processing dependencies for Scrapy
Searching for lxml
Reading http://pypi.python.org/simple/lxml/
Reading http://codespeak.net/lxml
Best match: lxml 3.0.1
Downloading http://lxml.de/files/lxml-3.0.1.tgz
Processing lxml-3.0.1.tgz
Running lxml-3.0.1\setup.py -q bdist_egg --dist-dir c:\users\t\appdata\lo
cal\temp\easy_install-7mgdch\lxml-3.0.1\egg-dist-tmp-686xm8
Building lxml version 3.0.1.
Building without Cython.
ERROR: 'xslt-config' is not recognized as an internal or external command,
operable program or batch file.

** make sure the development packages of libxml2 and libxslt are installed **

Using build configuration of libxslt
warning: no files found matching '*.txt' under directory 'src\lxml\tests'
In file included from src\lxml\lxml.etree.c:254:0:
c:\users\t\appdata\local\temp\easy_install-7mgdch\lxml-3.0.1\src\lxml\inc
ludes/etree_defs.h:9:31: fatal error: libxml/xmlversion.h: No such file or direc
tory
compilation terminated.
error: Setup script exited with error: command 'gcc' failed with exit status 1
Community
  • 1
  • 1
tomasz74
  • 16,031
  • 10
  • 37
  • 51
  • Apparently, you need to install the development packages of libxml2 and libxslt. – pydsigner Nov 04 '12 at 01:34
  • I installed them already – tomasz74 Nov 04 '12 at 01:35
  • Have you read http://doc.scrapy.org/en/latest/intro/install.html ? – pydsigner Nov 04 '12 at 01:36
  • yes I did, I followed all of it and installed dependecies – tomasz74 Nov 04 '12 at 01:37
  • Have you tried running a search on your C: drive for `xslt-config`? Perhaps you have it and need to add a folder to your path. – pydsigner Nov 04 '12 at 01:38
  • just searched for `xslt-config` but seems it does not exist. What should I do? But I have `libxslt` in `C:\Python27\Lib\site-packages` – tomasz74 Nov 04 '12 at 01:42
  • 1
    Try the binaries here: http://www.zlatkovic.com/libxml.en.html – pydsigner Nov 04 '12 at 01:49
  • For 64 bit Python [these instructions](http://steamforge.net/wiki/index.php/How_to_Install_Scrapy_in_64-bit_Windows_7) should work. – Pedro Romano Nov 04 '12 at 09:26
  • @pydsigner I went through the website you proposed. I downloaded it unziped, added paths to `bin` as it was suggested and when I run `easy_install -U Scrapy` I have all the same error. @Pedro Romano I run 32bit Windows I tried to go through it but probably would mess up more, so did not follow these instructions – tomasz74 Nov 04 '12 at 12:33
  • For everyone else who will have the same problem. I don't really know why it did not work on the first place, but after tries I succeeded. Problem was with `lxlm` I tried different ways but downloading `lxml` from http://www.lfd.uci.edu/~gohlke/pythonlibs/ worked after a few tries. Just make sure there is a folder `C:\Python27\Lib\site-packages\lxml` after running .exe. Before I also run `easy_install -U Cython`. On the end Scrapy was installed. – tomasz74 Nov 04 '12 at 13:19

2 Answers2

4

Not sure if you have solved it already. I ran into this problem today and here is how I solved it, grab the lxml binary directly: http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml

and pip install Scrapy somehow ignores the fact that you have it installed and tries to build again, so it will fail. However, in the directory where you run pip install there will be a 'build' directory, find 'Scrapy\setup.py' there.

Now find at the bottom of the file:

setup_args['install_requires'] = ...

and remove lxml and pyopenssl (you will most likely need to install the binary directly if you are working on windows, which btw can be found here: http://slproweb.com/products/Win32OpenSSL.html)

run pip install Scrapy again from the same directory and you should be good to go.

airfang
  • 1,238
  • 3
  • 14
  • 22
  • Did it work for you Tomas? If so can you please approve the answer? – airfang Nov 28 '12 at 02:59
  • This should have worked but I'm not sure why it didn't for me. pip kept attempting to reinstall lxml. It could be my virtualenv setup. In any case, I used ```pip install --no-deps Scrapy``` as an alternative. – lastoneisbearfood Jan 24 '13 at 07:29
  • On second thought, it probably would have worked if I copied lxml from my main python's site-packages to the virtualenv's site-packages first. – lastoneisbearfood Jan 24 '13 at 07:37
0

I had the same problem (Windows 10, Python 2.7.12) and here's what worked for me. I used "pip install scrapy" instead of "easy_install". If you don't have pip installed for 2.7, here's the official documentation. If that still doesn't work you can try following these manual installation directions

Greg Murray
  • 31
  • 6