Scrapinghub scrapy: ModuleNotFoundError: No module named 'pandas'

Question

I have tried deploying to Zyte via command line and GitHub but I have been stuck with the above error.

I have tried different versions of Scrapy version 1.5 to 2.5 but the error still persists.

I have also tried setting my Scrapinghub.yml to the below script and it doesn't seem to resolve the error

project: 557222

stacks:
    default: scrapy:1.5-py3






 Deploy log location: C:\Users\Public\Documents\Wondershare\CreatorTemp\shub_deploy_xzbjx9qc.log
Error: Deploy failed: b'{"status": "error", "message": "Internal error"}'
  File "/usr/local/lib/python3.6/site-packages/sh_scrapy/crawl.py", line 111, in _run_scrapy
    execute(settings=settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/cmdline.py", line 149, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/crawler.py", line 249, in __init__
    super(CrawlerProcess, self).__init__(settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/crawler.py", line 137, in __init__
    self.spider_loader = _get_spider_loader(settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/crawler.py", line 336, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "/usr/local/lib/python3.6/site-packages/scrapy/spiderloader.py", line 61, in from_settings
    return cls(settings)
  File "/usr/local/lib/python3.6/site-packages/scrapy/spiderloader.py", line 25, in __init__
    self._load_all_spiders()
  File "/usr/local/lib/python3.6/site-packages/scrapy/spiderloader.py", line 47, in _load_all_spiders
    for module in walk_modules(name):
  File "/usr/local/lib/python3.6/site-packages/scrapy/utils/misc.py", line 71, in walk_modules
    submod = import_module(fullpath)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
  File "/app/__main__.egg/greyhound_recorder_website/spiders/greyhound.py", line 7, in <module>
ModuleNotFoundError: No module named 'pandas'
{"message": "shub-image-info exit code: 1", "details": null, "error": "image_info_error"}
{"status": "error", "message": "Internal error"}

score 2 · Answer 1 · answered Oct 02 '21 at 02:37

2

You should specify the dependencies you need to be installed in Scrapy Cloud . eg.:

scrapinghub.yml:

project: 557222

stacks:
    default: scrapy:1.5-py3

requirements:
    file: requirements.txt

requirements.txt:

pandas

answered Oct 02 '21 at 02:37

Thiago Curvelo

3,711
1
22
38

score -1 · Answer 2 · answered Oct 01 '21 at 18:21

-1

Pandas is a python package like Scrapy. You need to install it. pip install pandas or visit https://pandas.pydata.org/docs/getting_started/install.html

answered Oct 01 '21 at 18:21

Damien

13
5

I have pandas installed on my environment, I don't think pandas installing is the problem – chuky pedro Oct 01 '21 at 18:22
Have you updated python and each package? Is pandas installed for python3.6? Did you used the same package management (pip, conda,...)? – Damien Oct 01 '21 at 19:04

Scrapinghub scrapy: ModuleNotFoundError: No module named 'pandas'

2 Answers2