-1

FIRST QUESTION: I was using python 2.7, but I started learning web scraping and the tutorial I read use python 3.5. So I installed python 3.5 and pip for that. After I installed scrapy I cannot run my project using

scrapy crawl mywork

it reports an error like this:

ImportError: cannot import name '_win32stdio'

And then I read that python 3.5 with scrapy is now not supported on windows, but Ubuntu. So I want to ask if there is any configuration that I didn't know?

SECOND QUESTION: And in order to avoid the problem I want to go back to python 2.7 and installing scrapy for 2.7. It is not hard to differentiate pip for 2.x and 3.x version by

py -2 -m pip isntall thePackage

But the PROBLEM is that after I installed python 3.5 and pip for 3.5, I cannot use pip for py2.7 anymore. It seems that I cannot link to the url for python 2.7 or something like that.

Whatever package I am installing in 2.7, it goes like 

Could not fetch URL https://pypi.python.org/.../: There was a problem 

confirming the ssl certificate: [Erron 185090050] _ssl.c:344 ...
....
Cannot fetch index base URL https://pypi.python.org/simple/
....
Could not find any downloads that satisfy the requirement pip in c:\python27\lib\site-packages\pip-1.5-py2.7.egg

I do love python but the installing pakcages part is really annoying

Xiaoyi.Zhang
  • 3
  • 3
  • Possible duplicate of [How to run multiple python version on Windows](http://stackoverflow.com/questions/4583367/how-to-run-multiple-python-version-on-windows) – glls May 24 '16 at 00:44
  • Thanks, I've read this one before but I dont think it resolves my question, i'm asking why pip not working when switching back to python2.7, and how to use scrapy in python3.5 in windows. – Xiaoyi.Zhang May 24 '16 at 01:42
  • have you checked your environment variables? – glls May 24 '16 at 01:50
  • the pip for my old version of python 2.7.8 works fine until I installed python3.4. I think that python3,4 might change some configuration in python2. Anyway, I uninstalled python2.7.8 and installed python2.7.11 because I read that pip is automatically installed since 2.7.11 version. – Xiaoyi.Zhang May 24 '16 at 19:40
  • Then, I started to install twisted which is a package required for scrapy. And I got the "valueError: [u'path']". I then checked if the same problem happens in both py2 and py3 when installing different packages. And I found that the Error only happens when I installing c-required packages in py2. I read a question in stackoverflow saying that I should install pywin32 or vs2008(with x64 Compiler and Tools, for the x64 system). But I've already have pywin32 in py2 and it doesn't help. And I just happened to have a vs2010 in my laptop, so I installed it and restarted my computer. – Xiaoyi.Zhang May 24 '16 at 19:49
  • And then, I can install twisted smoothly. After that, I tried to run scrapy in cmd and it raised a new error asking for pyasn1. To make sure that there was no problem with the scrapy itself I used pip to upgrade scrapy in py2 and after doing that everything works fine. And from the log it seems that pyasn1 is also a dependency for scrapy.ersion. – Xiaoyi.Zhang May 24 '16 at 19:49

2 Answers2

0

Regarding your issue using pip, you need to check and modify your environment variables:

https://docs.python.org/2/using/windows.html

3.3.1. Excursus: Setting environment variables Windows has a built-in dialog for changing environment variables (following guide applies to XP classical view): Right-click the icon for your machine (usually located on your Desktop and called “My Computer”) and choose Properties there. Then, open the Advanced tab and click the Environment Variables button.

In short, your path is:

My Computer ‣ Properties ‣ Advanced ‣ Environment Variables In this dialog, you can add or modify User and System variables. To change System variables, you need non-restricted access to your machine (i.e. Administrator rights).

Otherwise, remove pip and reinstall, as it might be related to the pip version you have (pip connection failure: cannot fetch index base URL http://pypi.python.org/simple/)

as per why scrapy is not woking using python 3.5 - this questionn has already been answered here: Error installing scrapy in python 3.5

and from a comment in this post:

Until Twisted supports Python 3.5 which it does not it will not work on windows.

– @Chris Hawkes

Community
  • 1
  • 1
glls
  • 2,325
  • 1
  • 22
  • 39
  • Thank you for your help. I've solved the problem now by reinstalling python 2.7. I was using python 2.7.8, and I switched into 2.7.11 which has pip installed automatically, and then I installed visual studio 2010. And then I used pip to install twisted, and upgrade scrapy and everything works fine. – Xiaoyi.Zhang May 24 '16 at 19:35
0

(Based on this answer) You can install Twisted on Python 3.5 by using the whl from here. Then try installing Scrapy again.

Hanch
  • 35
  • 4