0

When I use 

import psycopg2

in Scrapy pipeline.py or spider.py I get the following error

ModuleNotFoundError: No module named 'psycopg2'

I know that the error is not valid, because I am running Django server with postgresql, using same venv which contains psycopg2. And also I can connect to db in a test file and enquiry as well. In addition I can run import psycopg2 in Scrapy shell with no issue. So it has puzzled me that I might be missing something, I appreciate if someone can help me,

Nabat Farsi
  • 840
  • 1
  • 9
  • 17

2 Answers2

0

Since it took me hours, here is what I have found.

I tested both in windows and Linux Ubuntu, both had same error.

Apparently this is what is happening:

In windows I had Scrapy installed on Core, and in Ubuntu apparently Scrapy is already wrapped in.

I ran my crawler within venv, and it appeared that actually my Scrapy is not installed correctly in the venve. However, Since Scrapy was already in the PATH it was running from Core, where it was no psycopg2. So the above mentioned errors was actually come from come not venv.

Installing Scrapy looked like was very simple in venv at first. I checked that by trying Scrapy shell and it was running, but it was actually were running from the core again. Make sure that you have Scrapy in your pip freeze list in your venv. This is a link that has explained how to address some issues in installation.

Nabat Farsi
  • 840
  • 1
  • 9
  • 17
0

You need to install psycopg2 in your virtualenv, issue the next command

pip install psycopg2

Justo
  • 121
  • 3