Crawl website every x minutes/seconds using scrapy

Question

I'm crawling a blog using Scrapy using the command "scrapy crawl posts". I need to check for new posts every 5 minutes. How can I do it?

Thanks in advance.

@user6712306 this not code sharing site plz put your try code after you need help then ask freely — Divyesh Kanzariya, Nov 21 '16 at 17:23
Hook up to a Cron job for more info : http://newcoder.io/scrape/part-5/ — Divyesh Kanzariya, Nov 21 '16 at 17:27

score 1 · Answer 1 · edited May 23 '17 at 12:30

1

Edit your /etc/crontab file to include this line. It will setup your script to run every 5 minutes.

*/5 *   * * *   unix-username   cd /your/path/to/scrapy-project/;scrapy crawl posts

For windows you can take a look at the answer mentioned here

edited May 23 '17 at 12:30

Community

answered Nov 21 '16 at 17:17

1 Answers1