crontab task for scrapy

Question

I have configured a the scrapy task at crontab but it not running. however when i run manually it works. here is what i have tried.

* * * * * my_user cd /var/www/html/path/spiders/ && scrapy crawl radio

And here is my "radio" scrapy file with this permissions.

   -rwxr-xr-x 1 my_user  my_user 161 mar 27 10:09 __init__.py
   -rw-r--r-- 1 my_user  my_user 148 mar 28 01:00 __init__.pyc
   -rwxrwxr-x 1 my_user  my_user  2123590 abr 20 11:51 log.txt
   -rwxrwxr-x 1 my_user  my_user  5682 abr  3 09:58 radio.py
   -rwxrwxr-x 1 my_user  my_user  6029 abr  4 01:00 radio.pyc

Possible duplicate of [Scrapy crawler in Cron job](http://stackoverflow.com/questions/17235101/scrapy-crawler-in-cron-job) — Casper, Apr 20 '17 at 11:07
It depends whether SCRAPY is in PATH for Cron User or not ... — Umair Ayub, Apr 21 '17 at 07:41

score 1 · Answer 1 · answered Apr 21 '17 at 07:44

Login to your server as root

run echo $PATH

It will output something like this.

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

Now edit the crontab -e and put this at start of file.

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

Also run this command to see if Cron ran or not.

grep CRON /var/log/syslog

crontab task for scrapy

1 Answers1