0

I am trying to install scrapy and for that I have installed python 2.7 and conda(as pip was giving me errors that i couldn't solve.) after that I created a separate environment for scrapy named py2 with conda.

With this output I suppose that scrapy is installed successfully on my machine.

[py2] C:\Users\ADMIN>conda install -c https://conda.anaconda.org/scrapinghub scrapy
Fetching package metadata: ......
Solving package specifications: ...........................
# All requested packages already installed.
# packages in environment at C:\dev\Anaconda3\envs\py2:
#
scrapy                    1.0.3                    py27_2    scrapinghub

Now I make it sure that it is python 2.7 I executed this

[py2] C:\Users\ADMIN>py
Python 2.7.11 (v2.7.11:6d1b6a68f775, Dec  5 2015, 20:40:30) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

Which is also correct I suppose.

Now still I am unable to do this

>>> import scrapy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named scrapy

which indicates that scrapy is not installed.

Environment : Windows 8 (64 bit) I am a beginner both in python and scrapy.

Himanshu Bhandari
  • 1,769
  • 2
  • 23
  • 37

2 Answers2

1

There are interestingly a few hacks that are needed to run Scrapy on Windows.

You would need to install unofficial support for Windows in Python, look for PyWin32 section: http://www.lfd.uci.edu/~gohlke/pythonlibs/ Afterwards remember to run postinstall.py -install as advised on this page.

Depending how you are setup, you might need to install C++ compiler for Python, required to run lxml: https://www.microsoft.com/en-us/download/details.aspx?id=44266

Please look for similar questions referring to overall Python not only Scrapy e.g. error: Unable to find vcvarsall.bat Depending on how you are setup you will find hints around.

Community
  • 1
  • 1
Turo
  • 1,537
  • 2
  • 21
  • 42
  • As far as I know, wouldn't conda install all the dependencies when you create a new environment? In my case when I created a new environment followed by installing scrapy I have found that lxml is get installed. I have also installed "pywin32-220.win-amd64-py3.5.exe" however this is for python 3 and I don't have any idea how to get install the same for Python 2. Regarding "vcvarsall.bat", yes I got this error earlier but with help of SO I resolved that. * I require both Python versions on the same machine. – Himanshu Bhandari Feb 05 '16 at 05:17
  • 1
    follow the link I sent you for unofficial PyWin32, not the official one. If you scroll to PyWin32 you will find there pywin32-220-cp27-none-win32.whl, this cp27 indicates it is for Python 2. Also C++ compiler is needed because lxml depends on C++ , maybe you can fire Scrapy without it but then you might face issues with parsing. Go to Control Panel - System and Security - System - Advanced, System Settings - Environment Variables and ensure you have Python there C:\Python27\;C:\Python27\Scripts\; You can add there Python3 in the same way, the first version listed gets priority. – Turo Feb 05 '16 at 06:27
  • I am using 64 bit machine so instead of "pywin32-220-cp27-none-win32.whl" I suppose that I should go with "pywin32-220-cp27-none-win_amd64.whl" Right? I have also installed C++. for now I have installed the .whl file with pip install and it gets successfully installed after than i downloaded the script from http://ipython.scipy.org/misc/pywin32_postinstall.py and in py2 conda environment executed this. but still import scrapy doesn't resolved. :( – Himanshu Bhandari Feb 05 '16 at 07:23
  • I have tried to create a project and I got it working. "maybe you can fire Scrapy without it but then you might face issues with parsing." Now for parsing... It is still far away for me. But still I am unable to import scrapy. Don't know what is going on. – Himanshu Bhandari Feb 05 '16 at 07:31
  • If you indeed created a project in Scrapy, then Scrapy is installed on your machine. import scrapy is not a valid console comment neither for scrapy nor for Python, that's why get an error. Read the Scrapy docs.Type e.g. : scrapy shell http://stackoverflow.com in the console, if you get a response, means scrapy works. If not, read the stuff for install I recommended you. If you found any of these useful kindly upvote/accept the answers. Good luck! After few days of headache with Scrapy docs, you will get there :) – Turo Feb 05 '16 at 08:47
  • Hey thanks a lot for all the info and your valuable time. I found the solution and posting it as a answer. – Himanshu Bhandari Feb 05 '16 at 12:11
0

As it was all in virtual environment of conda, while installing scrapy I needed to pass the name of conda virtual environment, except that scrapy would have been installed in the root(I mean outside the environment or for the machine.)

So for py2 virtual environment, after activating this, following command will install scrapy:

conda install -n py2 -c https://conda.anaconda.org/scrapinghub scrapy

Here py2 is environment's name.

This page helped me to figure out what was happening here.

I must say python is not that easy to learn.

Himanshu Bhandari
  • 1,769
  • 2
  • 23
  • 37