Scrapy 2.5 - How do I associate my spiders with a specific feed export?

Question

How do I link a specific Spider with a specific Feed Export?

I currently have two spiders spider1 and spider2. I added the FEEDS dict to settings.py but am not sure how to associate the individual spiders to an individual feed URI / export.

I would use custom settings as noted here, but this method is deprecated in use of FEEDS.

Settings for FEEDS

FEEDS = {
    'csv/<exportfilename-spider1>.csv':{
        'format': 'csv',
        'overwrite': True
    },
    'csv/<exportfilename-spider2>.csv':{
        'format': 'csv',
        'overwrite': True
    }
}

score 1 · Answer 1 · answered Oct 10 '21 at 21:14

I ended up using the custom_settings in each individual spider. Please let me know if there is a better solution.

In yourSpiderName.py add:

    custom_settings= {
        'FEEDS': {
            'csv/<filename>.csv':{
                'format': 'csv',
                'overwrite': True
            }
        }

Scrapy 2.5 - How do I associate my spiders with a specific feed export?

1 Answers1