Cannot setup Scrapy on windows

Question

I tried to setup Scrapy on Windows 7 by steps described at http://doc.scrapy.org/en/latest/intro/install.html . On my PC was installed Python 3.5.1. Although Scrapy not support this python version it was installed successfully with latest Anaconda but fails to run spider script. I find that Scrapy only works with Python 3.3.+ version so uninstall version 3.5.1, uninstall Anaconda, install python 3.3.5, install pywin32 and install pip. pip fails pip install Scrapy, so I install Anaconda and run conda install -c scrapinghub scrapy Scrapy installed, but I saw that libs installed was for python 3.5 like: scrapy: 1.1.0-py35_0

Now I run the

c:\python\olxscrapy>scrapy crawl OlxCatalogSpider

and get error

  File "C:\Anaconda3\lib\site-packages\twisted\internet\stdio.py", line 30, in
module>
    from twisted.internet import _win32stdio
ImportError: cannot import name '_win32stdio'

How to make Scrapy run with python 3.3.+

hyber mat · Answer 1 · 2016-06-15T07:23:25.947

2

On this blog:

https://blog.scrapinghub.com/2016/05/25/data-extraction-with-scrapy-and-python-3/

it says Scrapy on Python 3 doesn't work in Windows environments yet

Edit: I recently installed scrapy on Ubuntu for Python 3.5 and received a lot of errors. The errors stopped after: "sudo apt-get install python3.5-dev".

edited Jun 15 '16 at 07:23

answered Jun 14 '16 at 17:36

hyber mat

21
6

1

Yes, but on FAQ page there is no any info about that: What Python versions does Scrapy support? Scrapy is supported under Python 2.7 and Python 3.3+. Python 2.6 support was dropped starting at Scrapy 0.20. Python 3 support was added in Scrapy 1.1. – P_M Jun 14 '16 at 17:44
Here link http://doc.scrapy.org/en/latest/faq.html . How to know is this blog post not outdated? Or shift to linux only solution? – P_M Jun 14 '16 at 17:46
Will Ubuntu or Debian work with Scrapy and Python 3.3.+? – P_M Jun 14 '16 at 17:59
@Pavlo, I don't know about your target Python 3.3 version. But Scrapy 1.1.0 was tested with Python 3.3.5 on Travis CI. See https://travis-ci.org/scrapy/scrapy/jobs/129470987 – paul trmbrth Jun 14 '16 at 18:04

score 1 · Answer 2 · answered Jul 18 '16 at 17:16

1

I add the follow package and it works:

pip install twisted-win==0.5.5

answered Jul 18 '16 at 17:16

Roberto Nunes

609
5
4

score 0 · Answer 3 · answered Jun 14 '16 at 18:56

0

Try to create a virtual env:

pip install virtualenv (instalation) virtualenv -p python3.3.5 envName (creation with specific python version) source ./envName/bin/activate (activate virtual env)

This way you can guarantee that's the right python version. Also scrapy has some requirements that can't be installed via pip and this may cause your pip install scrapy to fail

So install at your computer: python-dev libxslt1-dev libxslt1.1 libxml2-dev libxml2 libssl-dev

After this you finaly be able to install scrapy via pip inside your virtual env (probably)

Sry for my poor English isn't my native lang. Hope this work =]

answered Jun 14 '16 at 18:56

Vinicius de Castro

53
1
1
5

Thanks, for your efforts. I not familiar with virtual environment, so will check this approach. – P_M Jun 15 '16 at 05:35
I googled a bit, but looks like this is not what I need. I was able to setup python 3.3 but it just not compatible with scrapy on windows. – P_M Jun 15 '16 at 10:35

score 0 · Answer 4 · answered Aug 14 '17 at 05:56

Installation of Scrapy on Windows may facing error while installing Twisted.

Download Twisted according to your Python and windows version on this site http://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted
Turn to your download folder and pip install <downloaded filename>
pip install scrapy

Cannot setup Scrapy on windows

4 Answers4

Linked