scrapyd: How to include files into a deployd package

Question

I am able to run a crawler localy which reads some input from a local file inside the scrapy project. Deployment with scrapyd-deploy failes, as the local file is somehow not in the package.

inside scrapy project, open a file:

with open('imports/filter.csv', newline='') as f:
    for row in csv.reader(f):

setup.py:

from setuptools import setup, find_packages

setup(
    name         = 'test',
    version      = '1.0',
    packages     = find_packages(),
    entry_points = {'scrapy': ['settings = crawler.settings']},
    include_package_data = True,
    package_data={'': ['imports/*.csv']}
)

The package_data and include_package_data somehow does not have any effect:

 in GetbidSpider\nFileNotFoundError: [Errno 2] No such file or directory: 'imports/filter.csv'\n"}

How can I include files inside the project without using absolute paths?

Have you already tried [removing `include_package_data=True`](https://stackoverflow.com/a/13783919/9003106) ? — carpa_jo, May 08 '20 at 09:16
yes, I tried but it did not solve it. Added an answer that fixed it. — merlin, May 08 '20 at 19:01

score 0 · Answer 1 · answered May 08 '20 at 19:00

This solved my problem:

package_data={
    'crawler': ['imports/*.csv', 'resources/*',]
},    
entry_points = {'scrapy': ['settings = crawler.settings']},
zip_safe=False,

In combination with changing the way I read the files:

import pkgutil
MatFile = pkgutil.get_data('crawler', 'LILAC_Materials.csv')
dataReader = csv.reader(MatFile.decode('utf-8').splitlines(), delimiter=',')

scrapyd: How to include files into a deployd package

1 Answers1