Make scrapy export to csv

Question

I want to use scrapy in the following way

from scrapy.crawler import CrawlerProcess

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)' })

process.crawl(my_super_scraper) 
process.start()

It works with my_super_scraper, but I cannot figure out how to export to CSV. I cannot find it in the documentation either.

score 3 · Accepted Answer · edited Apr 29 '19 at 02:40

You need to set FEED_FORMAT and FEED_URI parameters as follow:

from scrapy.crawler import CrawlerProcess

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)', 
    'FEED_FORMAT': 'CSV', 
    'FEED_URI': 'file:///tmp/export.csv',
})

process.crawl(my_super_craper)
process.start()

More information about feed export here https://docs.scrapy.org/en/latest/topics/feed-exports.html

This post shows how to export to JSON format: Scrapy process.crawl() to export data to json

Make scrapy export to csv

1 Answers1