14

I met a very strange issue, running Scrapy with PyCharm:
With the exact same configuration, Debug works, but Run with PyCharm does NOT work.

  • Windows 10
  • PyCharm 2016.3.3
  • Scrapy 1.3.3
  • Python 3.6.0

Configuration: PyCharm Edit Configuration Page

When I attempt to debug scrapy, it works perfectly. When I attempt to run scrapy, I get below error:

C:\Users\baib2\AppData\Local\Programs\Python\Python36\python.exe 
C:/Users/baib2/AppData/Local/Programs/Python/Python36/Lib/site-packages/scrapy/cmdline.py crawl scenelist_spider
Traceback (most recent call last):
  File "C:/Users/baib2/AppData/Local/Programs/Python/Python36/Lib/site-packages/scrapy/cmdline.py", line 8, in <module   
    import scrapy
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\scrapy\__init__.py", line 27, in <module   
    from . import _monkeypatches
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\scrapy\_monkeypatches.py", line 20, in <module   
    import twisted.persisted.styles  # NOQA
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\twisted\persisted\styles.py", line 21, in <module   
    from twisted.python.compat import _PY3, _PYPY
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\twisted\python\__init__.py", line 11, in <module   
    from .compat import unicode
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\twisted\python\compat.py", line 612, in <module   
    from http import cookiejar as cookielib
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\Lib\site-packages\scrapy\http\__init__.py", line 8, in <module   
    from scrapy.http.headers import Headers
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\scrapy\http\__init__.py", line 10, in <module   
    from scrapy.http.request import Request
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\scrapy\http\request\__init__.py", line 8, in <module   
    from w3lib.url import safe_url_string
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\w3lib\url.py", line 17, in <module   
    from six.moves.urllib.request import pathname2url, url2pathname
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\six.py", line 92, in __get__
    result = self._resolve()
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\six.py", line 160, in _resolve
    module = _import_module(self.mod)
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\site-packages\six.py", line 82, in _import_module
    __import__(name)
  File "C:\Users\baib2\AppData\Local\Programs\Python\Python36\lib\urllib\request.py", line 88, in <module   
    import http.client
ModuleNotFoundError: No module named 'http.client'

Process finished with exit code 1

No need to attach any code because even running the command:

python3.exe C:\Users\baib2\AppData\Local\Programs\Python\Python36\Lib\site-packages\scrapy\cmdline.py

will produce the same error.

I've checked my sys.path, comparing with run, debug has 1 more path:

'C:\Program Files (x86)\JetBrains\PyCharm 2016.3.3\helpers\pydev'

And I really don't think this shall make any difference.

Hope someone can take a look, thanks!

JosephGarrone
  • 4,081
  • 3
  • 38
  • 61
Bai Bing
  • 381
  • 2
  • 15
  • `No module named 'http.client'` means you are running something that is meant for python3 on python2. – Granitosaurus Apr 07 '17 at 20:50
  • Thanks for your reply! But I'm 100% sure I'm using python3. You can see I'm using Python3 Interpreter in my attachment. – Bai Bing Apr 08 '17 at 03:52
  • 3
    Have you managed to solve this in the end ? I'm facing EXACTLY same symptoms - scrapy runs fine in debug, but fails on http.client when in normal run or from command line. – Lili Feb 02 '18 at 23:12
  • 1
    I didn't figured out the reason. I tried to enable "Show Command line afterwords", it worked. – Bai Bing Feb 04 '18 at 12:01
  • I had the same exact issue and surprisingly the solution worked. I still don't understand how it fixed it. – adarsh hegde Mar 06 '18 at 16:40
  • For me checking "Run with Python console" worked. I don't have Show Command line afterwards in my version of Pycharm (2018.3.5) – Shani Shalgi Mar 27 '19 at 16:50

3 Answers3

10

Menu Run>Edit configurations enable checkbox "Show Command line afterwards" This helps for me

Алекс Денькин
  • 301
  • 4
  • 9
  • Is there an explanation why this is the case? I just ran into the issue and it was resolved in the same way, but I still would like to know why it happens. – Jascha Grübel Jul 16 '18 at 13:05
5

In case you are using a new version of PyCharm where Show Command line afterward is not available anymore, Run with Python console solves the issue.

cozyss
  • 1,290
  • 1
  • 15
  • 22
  • 1
    This ends up in PyCharm to stuck on the task "Connecting to console...". It never finishes therefore the actual Python program is never started. – Robert Jan 09 '19 at 15:58
0

I've also spent some time tracking a similar problem.

My issue was that I have a local module named http which of course does not submodule called client. Another similar issue is described here.

The solution is to change Working directory for this test invocation via Edit configurations.

tworec
  • 4,409
  • 2
  • 29
  • 34