1

I am trying to debug scrapy with pycharm, when scrappy is run from a conda virtual environment.

I followed this link How to use PyCharm to debug Scrapy projects and it normally works, but if I use scrapy from a conda virtual environment I get

import http.client ModuleNotFoundError: No module named 'http.client'

In a "regular" (venv) environment it works perfectly.

I checked and the virtual environment does have http.client. Also other libraries from same folder (ex hashlib.py) are imported.

this is the code in the file I am running

from scrapy import cmdline
cmdline.execute("scrapy crawl my_spider".split())

I already tried adding the virtual environment's path to PYTHONPATH

Update: I added steps on how to reproduce this error:

  1. create a conda environment conda create -n testenv1 python=3.7

  2. install scrapy and create a project and spider pip install scrapy cd D:\Learn\scrapy scrapy startproject test1 cd test1 scrapy genspider googspid google.com

  3. Create a pycharm project in scrapy project's root (where scrapy.cfg is) and choose python.exe from the virtual env just created as interpreter

  4. Follow the best answer from link above and you'll get this error ModuleNotFoundError: No module named 'http.client'

emo1
  • 11
  • 5

1 Answers1

0

If you're creating a PyCharm Run Configuration for running Scrapy, that Run Configuration needs to use the same virtual environment; you can do this by going to Project Settings > Python Interpreter and finding (or manually adding) the virtualenv's Python executable as the Python interpreter for the project (or for that specific run configuration, if you prefer).

Avish
  • 4,516
  • 19
  • 28