Cannot deploy spiders with scrapyd deploy (shows 0 spiders)

Question

While attempting to deploy spiders to scrapyd running locally, I am getting the following response:

{"node_name": "a-38u3442zr18hl", "status": "ok", "project": "project_name", "version": "1662563564", "spiders": 0}

I have 6 spiders which are running fine with the scrapy runspider command.

I'm running Scrapy version 2.6.2, and all dependencies are installed in the virtual environment, which is also where I'm running scrapyd-deploy.

I saw here that scrapy list should return my spiders, but it returns nothing.

$ scrapy list
$

Settings file:

DATABASE = "database"
USER = "user"
PASSWORD = "password"
HOST = "host"
PORT = "port"
POSTGRES_CONNECTION_STRING = f"postgresql://{USER}:{PASSWORD}@{HOST}:{PORT}/{DATABASE}"

LOG_FILE="log.txt"
DOWNLOADER_MIDDLEWARES = {
    'project_name.middlewares.get_headers.GetHeaders': 901,
}
MEDIA_ALLOW_REDIRECTS = True
AWS_ACCESS_KEY_ID='aws_access_key_id'
AWS_SECRET_ACCESS_KEY='aws_secret_access_key'

S3_BUCKET = "s3_bucket"

scrapy.cfg file:

[settings]
default = project_name.settings

[deploy]
url = http://localhost:6800/
project = project_name

setup.py file:

# Automatically created by: scrapyd-deploy

from setuptools import setup, find_packages

setup(
    name         = 'project',
    version      = '1.0',
    packages     = find_packages(),
    entry_points = {'scrapy': ['settings = project_name.settings']},
)

Project's directory tree:

.
├── project_name
│   ├── __init__.py
│   └── spiders
│       ├── __init__.py
│       ├── first_spider.py
│       ├── second_spider.py
│       ├── ...
│       ├── sixth_spider.py
│   ├── items
│   │   ├── __init__.py
│   │   ├── first_item.py
│   │   ├── processors.py
│   ├── pipelines
│   │   ├── __init__.py
│   │   ├── first_pipeline.py
│   │   ├── second_pipeline.py
│   ├── models
│   │   ├── __init__.py
│   │   ├── models.py
│   ├── helpers
│   │   ├── __init__.py
│   │   ├── helpers.py
│   ├── middlewares
│   │   ├── first_middleware.py
│   ├── settings.py
├── scrapy.cfg
├── setup.py
├── Pipfile
├── Pipfile.lock
├── README.md

python version: 3.7.10
scrapyd version: 1.3.0
scrapyd-client version: 1.2.2
platform: Amazon Linux 2

How can I debug why the spiders are not being listed / deployed?

Cannot deploy spiders with scrapyd deploy (shows 0 spiders)

0 Answers0