Scrapy not recognised command

Question

I have successfully installed Scrapy and all it's dependencies into Python.org 64 shell on a Windows Vista 64 bit platform. This is installed as Site Package. I have tested Scrapy works ok at a basic level by typing:

Import Scrapy

Into a Python Shell and got no errors. I've just got started with tutorials on how to use Scrapy. Most of these suggest setting up a new project using the following Command Prompt line:

C:\Python27>scrapy startproject myproject

However this throws up the error message in the title about Scrapy being an invalid command. I have tried amending my environment variables to allow me to use this command from any file location within Command Prompt. I have added the following to the end of my environment variables:

C:\Python27;C:\Python27\Scripts;C:\Python27\Lib\site-packages\scrapy;C:\Python27\Scrapy

This does not seem to resolve the problem however as I still get the same error message. I have also tried running the 'scrapy startproject myproject' command from the following locations with Command Prompt:

C:\Python27
C:\Python27\Scripts
C:\Python27\Scrapy

This again still throws up the same error message. I feel like I am quite close to having the solution, but I just cant quite seem to make it work.

Can anyone spot where I am going wrong?

Thanks

EDIT:

In response to the last comment above from MattDMo, the command now works, however it now throws up the following error message, which I'm not even sure where to start with:

C:\Python27\Scripts>scrapy startprogram mrscript
Traceback (most recent call last):
  File "c:\Python27\Scripts\scrapy", line 3, in <module>
    from scrapy.cmdline import execute
  File "c:\Python27\lib\site-packages\scrapy\cmdline.py", line 9, in <module>
    from scrapy.crawler import CrawlerProcess
  File "c:\Python27\lib\site-packages\scrapy\crawler.py", line 3, in <module>
    from twisted.internet import reactor, defer
  File "c:\Python27\lib\site-packages\twisted\internet\reactor.py", line 38, in
<module>
    from twisted.internet import default
  File "c:\Python27\lib\site-packages\twisted\internet\default.py", line 56, in
<module>
    install = _getInstallFunction(platform)
  File "c:\Python27\lib\site-packages\twisted\internet\default.py", line 50, in
_getInstallFunction
    from twisted.internet.selectreactor import install
  File "c:\Python27\lib\site-packages\twisted\internet\selectreactor.py", line 1
8, in <module>
    from twisted.internet import posixbase
  File "c:\Python27\lib\site-packages\twisted\internet\posixbase.py", line 24, i
n <module>
    from twisted.internet import error, udp, tcp
  File "c:\Python27\lib\site-packages\twisted\internet\tcp.py", line 29, in <mod
ule>
    from twisted.internet._newtls import (
  File "c:\Python27\lib\site-packages\twisted\internet\_newtls.py", line 21, in
<module>
    from twisted.protocols.tls import TLSMemoryBIOFactory, TLSMemoryBIOProtocol
  File "c:\Python27\lib\site-packages\twisted\protocols\tls.py", line 41, in <mo
dule>
    from OpenSSL.SSL import Error, ZeroReturnError, WantReadError
  File "c:\Python27\lib\site-packages\OpenSSL\__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "c:\Python27\lib\site-packages\OpenSSL\rand.py", line 11, in <module>
    from OpenSSL._util import (
  File "c:\Python27\lib\site-packages\OpenSSL\_util.py", line 4, in <module>
    binding = Binding()
  File "c:\Python27\lib\site-packages\cryptography\hazmat\bindings\openssl\bindi
ng.py", line 87, in __init__
    self._ensure_ffi_initialized()
  File "c:\Python27\lib\site-packages\cryptography\hazmat\bindings\openssl\bindi
ng.py", line 106, in _ensure_ffi_initialized
    libraries=libraries,
  File "c:\Python27\lib\site-packages\cryptography\hazmat\bindings\utils.py", li
ne 80, in build_ffi
    extra_link_args=extra_link_args,
  File "c:\Python27\lib\site-packages\cffi\api.py", line 341, in verify
    lib = self.verifier.load_library()
  File "c:\Python27\lib\site-packages\cffi\verifier.py", line 75, in load_librar
y
    return self._load_library()
  File "c:\Python27\lib\site-packages\cffi\verifier.py", line 151, in _load_libr
ary
    return self._vengine.load_library()
  File "c:\Python27\lib\site-packages\cffi\vengine_cpy.py", line 138, in load_li
brary
    raise ffiplatform.VerificationError(error)
cffi.ffiplatform.VerificationError: importing 'c:\\Python27\\lib\\site-packages\
\cryptography\\_Cryptography_cffi_444d7397xa22f8491.pyd': DLL load failed: The s
pecified module could not be found.

Is this actually an error log or just a standard print out? Also, could someone explain please why adding a .bat file to my Python scripts causes this to work?

Thanks

Answer 1

pip install scrapy installs a file called scrapy in Python's Scripts directory on Windows, the default location for which is C:\Python27\Scripts. However, C:\Python27\Scripts\scrapy is a Python file, even though it lacks the .py suffix. There are two ways of running it. My preferred method is to use the Git Shell, as scrapy has its executable bit set, and can be run without further ado.

The other method, if you want to keep using cmd.exe, is to make a batch file with the following contents:

@echo off
c:\Python27\python c:\Python27\Scripts\scrapy %*

Save it in the Scripts directory as scrapy.bat (assuming your PATH now includes C:\Python27;C:\Python27\Scripts), restart cmd.exe, and you should now be able to run it from the command line.

Answer 2

I experienced a similar problem (I'm using a virtual environment in Windows 7).

I noticed that for older versions of scrapy, e.g. pip install scrapy==0.18.4, it automatically creates a scrapy.bat in my virtual environment: {VENV_ROOT_FOLDER}\Scripts\scrapy.bat

But if I use the latest scrapy (currently version 0.24.2), the scrapy.bat is missing.

I wanted to use the latest scrapy version, so what I did is to put a copy of scrapy.bat to its Scripts folder and it solves the problem.