1

Python 3 is not my default version. I want to use it, because one the package I want to use toripchanger is only avaible under Python3.

So my pip3 version is:

C:\Users\Truc>pip3 -V
pip 18.1 from c:\python\python37\lib\site-packages\pip (python 3.7)

I use a scrapy.spider that I used to run in python 2.7 version and it works perfectly, but know I am using 3.7 in oder to use toripchanger I precised in my files this at the top of it:

#!"C:\Python37\python.exe"
# -*- coding: utf-8 -*-

the code above concerns the spider's file itself, the middlewares.py, the settings.py, the items.py, and pipelines.py.

In the middlewares.py I added some lines:

...
from scrapy.conf import settings
from toripchanger import TorIpChanger
...
ip_changer = TorIpChanger(reuse_threshold=3)

class ProxyMiddleware(object):
    _requests_count = 0

    def process_request(self, request, spider):
        self._requests_count += 1
        if self._requests_count > 10:
            self._requests_count = 0 
            ip_changer.get_new_ip()

        request.meta['proxy'] = settings.get('HTTP_PROXY')
        spider.log('Proxy : %s' % request.meta['proxy'])

The issue when I run my spider scrapy crawl spider is I obtain this error:

File "C:\Users\Truc\...\middlewares.py", line 13, 
in <module> from toripchanger import TorIpChanger
ImportError: No module named toripchanger

You may see the whole lines of error at the end of this topic

The issue for me is I tried what it is provided in that topic so I did pip3 install setuptools even pip3 install --upgrade setuptools, I did pip3 install Twisted-18.9.0-cp37-cp37m-win32.whl as adviced in this topic because it was necessary to install scrapy in my python 3.7 version.

As I said, the whole lines of error, where I see c:\python27 while I precised #!"C:\Python37\python.exe" in my files

[in] scrapy crawl spider
[out]

2019-02-18 14:27:11 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot:I_AM_A_POLITE_ROBOT)

2019-02-18 14:27:11 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.5, 
cssselect 1.0.3, parsel 1.4.0, 
w3lib 1.19.0, Twisted 17.9.0, 
Python 2.7.9 (default, Dec 10 2014, 12:24:55) [MSC v.1500 32 bit (Intel)], 
pyOpenSSL 17.5.0 (OpenSSL 1.1.0g  2 Nov 2017), 
cryptography 2.1.4, Platform Windows-7-6.1.7601-SP1
2019-02-18 14:27:11 [scrapy.crawler] INFO: Overridden settings
{'NEWSPIDER_MODULE': 'folder.spiders', 
'ROBOTSTXT_OBEY': True, 'SPIDER_MODULES': ['folder.spiders'],
'RETRY_TIMES': 5, 'BOT_NAME': 'I_AM_A_POLITE_ROBOT',
'RETRY_HTTP_CODES': [401, 403, 404, 408, 500, 502, 503, 504],
'AUTOTHROTTLE_ENABLED': True,'DOWNLOAD_DELAY': 2}

2019-02-18 14:27:11 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.logstats.LogStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.throttle.AutoThrottle']

2019-02-18 14:27:11 [py.warnings] WARNING: 
c:\python27\lib\site-packages\scrapy\utils\deprecate.py:
156:ScrapyDeprecationWarning: 
`scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware` 
class is deprecated, use
`scrapy.downloadermiddlewares.useragent.UserAgentMiddleware`
instead ScrapyDeprecationWarning)
Unhandled error in Deferred:

2019-02-18 14:27:11 [twisted] CRITICAL: Unhandled error in Deferred:

2019-02-18 14:27:11 [twisted] CRITICAL:
Traceback (most recent call last):
File "c:\python27\lib\site-packages\twisted\internet\defer.py", line 1386, in
_inlineCallbacks
result = g.send(result)
File "c:\python27\lib\site-packages\scrapy\crawler.py", line 98, in crawl
six.reraise(*exc_info)
File "c:\python27\lib\site-packages\scrapy\crawler.py", line 80, in crawl
self.engine = self._create_engine()
File "c:\python27\lib\site-packages\scrapy\crawler.py", line 105, in _create_engine
return ExecutionEngine(self, lambda _: self.stop())
File "c:\python27\lib\site-packages\scrapy\core\engine.py", line 69, in __init__self.downloader = downloader_cls(crawler)
File "c:\python27\lib\site-packages\scrapy\core\downloader\__init__.py", line88, in __init__self.middleware = DownloaderMiddlewareManager.from_crawler(crawler)
File "c:\python27\lib\site-packages\scrapy\middleware.py", line 58, in from_crawler
return cls.from_settings(crawler.settings, crawler)
File "c:\python27\lib\site-packages\scrapy\middleware.py", line 34, in from_settings
mwcls = load_object(clspath)
File "c:\python27\lib\site-packages\scrapy\utils\misc.py", line 44, in load_object
mod = import_module(module)
File "c:\python27\lib\importlib\__init__.py", line 37, in import_module__import__(name)
File "C:\Users\...\middlewares.py", line 13, in <module>
from toripchanger import TorIpChanger
ImportError: No module named toripchanger
AvyWam
  • 890
  • 8
  • 28
  • how are you calling `scrapy`? – gold_cy Feb 18 '19 at 14:02
  • as aws_apprentice is hinting, if you do "python myfile" and your myfile has the hashbang of "#!"C:\Python37\python.exe"", your system will still use the default python (probably 2.7 for your system). The hashbang only works if the file is run directly from from a shell.. usually in *Nix systems like "./myfile.py". Windows probably ignores the #! notation. –  Feb 18 '19 at 17:01
  • You need to include the actual command you use to start the program. –  Feb 18 '19 at 17:05
  • Also windows will use the "%PATH%" variables to find executables for files (i.e. which path to find the python.exe to run *.py files.) and not the hashbang. You will need to change to path items to remove the C:/Python27 references or you will have to do it like this "C:\Python37\python.exe scapy.py".. also isn't the Windows exe file called wpython or pythonw or something?? –  Feb 18 '19 at 17:13

2 Answers2

0

Ok, just in case you installed Python 2.7 after 3.7, Windows will by default consider the latest installed Python version as the default version.

lalam
  • 195
  • 1
  • 11
  • No. My versions 3.7 is the lastest installed and it do not makes it the default version automatically, at least in my windows 7 version. – AvyWam Feb 19 '19 at 11:41
0

Windows will always use the default python.

https://docs.python.org/3/faq/windows.html#how-do-i-make-python-scripts-executable

To use some other python version without changing the Windows default, you must do something like this (the exact path will vary per system setup):

C:\Program Files\[your python path]\python.exe "myfile.py"
  • The issue is this solution means to make a script, and it is not the case of my spider. I use arguments in the command to makes differents versions of the spider like [this](https://stackoverflow.com/questions/52977185/how-to-run-several-versions-of-one-single-spider-at-one-time-with-scrapy). Besides to be honest, when I downloaded another version of python I managed to crawl it with the version 3, but it asked for a module win32 that couldn't be installed by pip because it was not available anymore it was a mess of issues. – AvyWam Feb 19 '19 at 11:40
  • So I tried what abccd suggest in [this topic](https://stackoverflow.com/questions/44074371/how-do-i-use-scrapy-in-different-versions-of-python) like this `py -m scrapy crawl spider -a arg1=machin -a arg2=bidule` it worked in the python 3 version I installed, and as I told you just 10 minutes ago I obtain `import win32api ModuleNotFoundError: No module named 'win32api'` and when I try to install it with `pip3 install win32api` I obtain `Could not find a version that satisfies the requirement win32api (from versions: ) No matching distribution found for win32api` – AvyWam Feb 19 '19 at 12:02
  • You need to change your question. You can either 1) change the default python version, -or- 2) launch the proper version of python some other way. (or provide more information on why you can't start your program; your comments are unclear). –  Feb 20 '19 at 22:45
  • At the present, your question indicates your issue is an import error; which is not the case: You are not running the correct python and your question is amounting to an endless back and forth on possible problems. –  Feb 20 '19 at 22:48