Automate custom Django management command with crontab, within virtualenv

Question

I am trying to run a custom management command on my Django app, installed under a virtualenvironment.

This task is supposed to run once every minute, and I cannot quite figure out why it doesn't work.

Here's what I have in my crontab:

*/1 * * * * /home/myuser/.virtualenvs/myvirtualenv/bin/python /home/myuser/myapp/manage.py mycommand --settings=myapp.settings

This won't work, even if I see the command executing once a minute in /var/log/syslog, but it must be failing because it produces no effect.

If I just copy and run the same command, everything goes smooth.

I then tried running instead an external bash script, which again perfectly works if I run it manually:

*/1 * * * * source /home/myuser/myapp/myscript

The script activates the virtualenv and runs the management command. Again, no results.

I am a bit lost, as I've searched around thoroughly and I am completely stuck, as apparently no error messages show up anywhere.

What logging goes your management command do? It'd be worth capturing as much as you can to see where it's blowing up, else you're effectively debugging while blind. — Steve Jalim, Mar 22 '16 at 20:08
Also what environment is this running on? VPS/shared hosting? — Steve Jalim, Mar 22 '16 at 20:09
My command does a fair share of logging, but I can't see any of it unless I run the command manually. This leads me to thinking that I must have screwed something up with cron, rather than the command. It is running on a VPS yes. — spookyjudges, Mar 23 '16 at 09:19

score 1 · Answer 1 · answered Mar 22 '16 at 20:22

1

The problem might be that myapp.settings isn't on your Python path. I would try changing directory first:

*/1 * * * * cd /home/mysuser/myapp && /home/myuser/.virtualenvs/myvirtualenv/bin/python manage.py mycommand --settings=myapp.settings

answered Mar 22 '16 at 20:22

Alasdair

298,606
55
578
516

Nope, that did not help. – spookyjudges Mar 23 '16 at 09:22
Redirect the stdout and stderr to a file e.g. append `>> /path/to/logfile 2>&1` to the command. Then you might see what the problem is in the log file. – Alasdair Mar 23 '16 at 09:38

score 0 · Answer 2 · answered Mar 22 '16 at 22:37

After fiddling with crontabs myself in one of my projects, I started to use Celery (more specifically Celery Beat) for my periodic tasks. It is only a little more complicated to set up (You will need a message broker e.g. Redis) but it's much more powerful afterwards.

Even if you do not use the asynchronous task execution, the ease of managing cron jobs is worth the setup. Jobs can simply be defined right in your project's code by the usage of a function decorator, no need to edit the crontab on the server.

For managing the Celery worker(s) and the celery beat process, I would recommend the usage of supervisord

I fear I will have to go that way as well. I spent some 2 days already trying to get this command run via cron, but it just doesn't seem the right way of doing it. Celery on the other hand was scaring me, exactly because of the reasons you mentioned. — spookyjudges, Mar 23 '16 at 09:20

score 0 · Answer 3 · answered Apr 03 '20 at 14:38

The reason there are no error messages showing up is because cron sends them by email to the postbox of the user whose crontab is being executed.

If you created the cronjobs with crontab -e, this will be the postbox of the user you are logged in with.
If you created the cronjobs with sudo crontab -e, this will be the postbox of the root user.

To read the emails you must set up an email server and client. For this, postfix (which is the server) and mutt (the client) are very popular options and their installations are covered in many good tutorials.

Make sure to differentiate between your logged-in user's postbox, which is opened via mutt and the root user's postbox, which needs sudo mutt.

Once you have this set up, debugging will be a lot easier.

score 0 · Answer 4 · edited Nov 28 '20 at 14:26

I had a similar problem. My code was not able to get any variables from the settings file. It was fixed when I changed directory and activated the virtual environment.

*/1 * * * * cd /home/myuser/myapp && source /home/myuser/.virtualenvs/myvirtualenv/bin/activate && /home/myuser/.virtualenvs/myvirtualenv/bin/python /home/myuser/myapp/manage.py mycommand --settings=myapp.settings

should work in your case.

Automate custom Django management command with crontab, within virtualenv

4 Answers4