How do I package a Scrapy script into a standalone application?

Question

I have a set of Scrapy spiders. They need to be run daily from a desktop application. What is the simplest way (from user's point of view) to install and run it on another windows machine?

Answer 1

Create a script (e.g. run_spider.py) which runs scrapy crawl <spider_name> as a system command.

run_spider.py

from os import system
output_file_name = 'results.csv'
system('scrapy crawl myspider -o ' + output_file_name + ' -t csv')

Then feed that script to PyInstaller:

pyinstaller run_spider.py

Answer 2

The simplest way is write a script in python for them I guess...

If you are running a Windows Server you can even schedule the comand that you use (scrapy crawl yoursprider) to run the spiders.

Answer 3

Here is another possibility to run your spider as standalone script or executable

import scrapy
from scrapy.crawler import CrawlerProcess

class MySpider(scrapy.Spider):
    # Your spider definition
    ...

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

process.crawl(MySpider)
process.start() # the script will block here until the crawling is finished

You can find more information here: https://doc.scrapy.org/en/1.0/topics/practices.html