9

I found about scrapy that is a great tool scraping so i tried to install scrapy on my machine, but when i tried to do pip install scrapy it installed for a while and threw me this error..

error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools

I tried to install it in virtual environment but still the problem persists.

EDIT: here is what i got after the error..

    error: Microsoft Visual C++ 14.0 is required. Get it with "Microsoft Visual C++ Build Tools": http://landinghub.visualstudio.com/visual-cpp-build-tools

    ----------------------------------------
Command "d:\pycharmprojects\environments\scrapyenv\scripts\python.exe -u -c "import setuptools, tokenize;__file__='C:\\Users\\User\\AppData\\Local\\Temp\\pip-build-arbeqlly\\Twisted\\setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record C:\Users\User\AppData\Local\Temp\pip-jdj93131-record\install-record.txt --single-version-externally-managed --compile --install-headers d:\pycharmprojects\environments\scrapyenv\include\site\python3.5\Twisted" failed with error code 1 in C:\Users\User\AppData\Local\Temp\pip-build-arbeqlly\Twisted\

any help?

P.hunter
  • 1,345
  • 2
  • 21
  • 45
  • 3
    did you try visiting the suggested url? – eLRuLL Apr 14 '17 at 10:07
  • yes i did, but the question is what does scrapy has to do with visual studio – P.hunter Apr 14 '17 at 10:12
  • I would assume scrapy is using an underlying C/C++ library for speed. I had thought if you installed via wheel, Windows would be prepackaged, but I could be mistaken. (Also note that http://stackoverflow.com/questions/32350375/error-code-1-installing-scrapy#32419233 has a similar issue for a Mac user) – Foon Apr 14 '17 at 11:58
  • The current recommended way to install Scrapy on Windows is to [use Anaconda/Miniconda with the conda-forge channel](https://docs.scrapy.org/en/latest/intro/install.html#windows). – paul trmbrth Apr 14 '17 at 12:23

8 Answers8

16

Install Using Conda

If you have conda Use: conda install -c anaconda scrapy

Or

Install Using pip

Download https://www.lfd.uci.edu/~gohlke/pythonlibs/#twisted at this location

//Download the whl file of same version as Python version here cp27 means python2.7
pip install Twisted-18.9.0-cp27-cp27m-win_amd64.whl 
pip install scrapy
Community
  • 1
  • 1
Sameer Kumar Choudhary
  • 1,515
  • 12
  • 13
  • 3
    Downloading the Twisted file and installing it with pip was definitely the fastest and simplest solution for me – NJS Mar 24 '19 at 21:07
  • Holy macaroni this does the trick. No need to download that 4 GB of visual studio. – Saharsh Apr 15 '19 at 18:20
9

UPDATE As some one pointed out in the edit, if you download only the vc_redist.x64.exe it's just 15MB, for that go to "other tools and frameworks" and choose MS Visual C++

solved the problem by installing the tools from here

the thing is that some python modules require C++ tools to compile with a python wrapper around, as they are written in C++.

once the setup is downloaded you can select the specific tools or download other facilities provided.

it's somewhere around 4 GB but solves the purpose if in future you need to use some other module which are made in C++.

P.hunter
  • 1,345
  • 2
  • 21
  • 45
3

but the question is what does scrapy has to do with visual studio

Some python modules require a C/C++ compiler to build. That's why you need Visual Studio.

Quoting from Scrapy Installation Guide:

some binary packages that Scrapy depends on (like Twisted, lxml and pyOpenSSL) require a compiler available to install, and fail if you don’t have Visual Studio installed.

Chankey Pathak
  • 21,187
  • 12
  • 85
  • 133
0

I have just installed the Visual Studio desktop tools for C++ and solve it.

Helio Junior
  • 153
  • 2
  • 15
0

Install and use anaconda (A Python packages manager) instead of pip.

Download anaconda from here then use:

conda install scrapy
mr L
  • 1,008
  • 13
  • 20
0

You have to install Python 2.7 then you have to put Python27 and Pyhton27/Scripts on path and install Visual C++ 14 or above and type pip install Scrapy in your cmd.

Abhishek Raj
  • 23
  • 3
0

Installing only the "vc_redist.x64.exe" didn't work for me. I downloaded Visual Studio 2019 Community here and then chose "C++ desktop developement" and then only the first two boxes (see the following screenshot ont the right):

enter image description here

This worked for me.

Pozinux
  • 964
  • 2
  • 10
  • 22
-1

Just installed visual studio build tools and the problem get solved. Happy coding!

SOUMAVA BISWAS
  • 11
  • 1