How to create a cron job in server for starting celery app?

Question

the server where I have hosted my app usually restarts due to maintenance and when it does a function that I kept open in the background stops and I have to manually turn it on.

Here are the commands that I do in ssh

ssh -p19199 -i <my ssh key file name> <my username>@server.net

source /home/myapp/virtualenv/app/3.8/bin/activate

cd /home/myapp/app

celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log --detach

I need to start this celery app whenever there is no 'celery' function in the command ps axuww. If it is running already then it will show:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
myapp  8792  0.1  0.2 1435172 82252 ?       Sl   Jun27   1:27 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp  8898  0.0  0.2 1115340 92420 ?       S    Jun27   0:32 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp  8899  0.0  0.2 1098900 76028 ?       S    Jun27   0:00 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp  8900  0.0  0.2 1098904 76028 ?       S    Jun27   0:00 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp  8901  0.0  0.2 1098908 76040 ?       S    Jun27   0:00 /home/myapp/virtualenv/app/3.8/bin/python3 -m celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log
myapp 28216  0.0  0.0  10060  2928 pts/1    Ss   15:57   0:00 -bash
myapp 28345  0.0  0.0  49964  3444 pts/1    R+   15:57   0:00 ps axuww

I need the cron job to check every 15 minutes.

Answer 1

You have everything figured out already, just put it inside a IF block in a shell (Bash) script, save it wherever you feel like (eg, $HOME). After the script, we'll set Cron.

I'm gonna call this script "start_celery.sh":

#!/usr/bin/env bash -ue
#
# Start celery only if it is not running yet
#
if [[ `ps auxww | grep "[c]elery"` ]]; then

  source /home/myapp/virtualenv/app/3.8/bin/activate
  cd /home/myapp/app
  celery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log --detach

fi

Remember to make it executable!

$ chmod +x start_celery.sh

A few notes:

• the output from ps is being filtered by grep "[c]elery" because we don't one the grep line that would come out if we simply did grep "celery" (reference: https://www.baeldung.com/linux/grep-exclude-ps-results).

  Using grep "celery" | grep -v "grep" would not work in our case because we want to use the lack of celery commands (exit code "1") to trigger the IF block.

• setting "-ue" in the script fails the whole thing/script in case something goes wrong (this is a precaution I always use, and I live it here as a shared knowledge/hint. Even if we are not using variables (-u) I like to use it). Feel free to remove it. (reference https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html)

Now we have the script -- ~/start_celery.sh in our home directory, we just have to include it in our (user's) crontab jobs. To do that, we can use some help from some websites of our (awesome) geek community:

• The first one is https://crontab.guru/ . If nothing else, it puts the cronjobs syntax in very simple, visual, straight words. They also offer a payed service in case you want to monitor your jobs.
• In the same spirit of online monitoring (maybe of your interest, I live it here for the records and because it's free), you may find interesting https://cron-job.org/ .

Whatever the resources we use to figure out the crontab line we need, "every 15 minutes" is: */15 * * * * . To set that up, we open our (user's) cronjob table (crontab) and set our job accordingly:

1. Open crontab jobs:

$ crontab -e

2. Set the (celery) job (inside crontab):

*/15 * * * * /home/myapp/start_celery.sh

This should work, but I didn't test the whole thing. Let me know how that goes.

Answer 2

I think what you need is a corn job running your script at startup as long as you don't experience any function crashes.

Start by opening a terminal window and run the command below. Don't forget to <add your ssh key> and <username>. This will generate the script.sh file for you and save it in $HOME

echo -e "ssh -p19199 -i <my ssh key file name> <my username>@server.net\n\nsource /home/myapp/virtualenv/app/3.8/bin/activate\n\ncd /home/myapp/app\n\ncelery -A app.mycelery worker --concurrency=4 --loglevel=INFO -f celery.log --detach" >> $HOME/startup.sh

Make it executable using the following command:

sudo chmod +x $HOME/startup.sh

Now create the cron job to run the script when you restart your server. In the terminal create a new cron job.

crontab -e

This will open a new editor screen for you to write the cron job. Press the letter i on your keyboard to start the insert mode and type the following

@reboot sh $HOME/startup.sh

Exit the editor and save your changes by pressing esc key then type :wq and hit Enter

Restart your server and the cron job should run the script for you.

Answer 3

If you want to manage the tasks that should run when a system reboots, shutdowns are alike, you should not make use of cron. These tasks are managed by the Power Management Utilities (pm-utils). Have a look at man pm-action and this question for a somewhat detailed explanation.

Obviously, as a user it is not always possible to configure these hooks for pm-util and alternative options are considered. Having a cron-job running every x minutes is an obvious solution that would allow the user to automatically start a job.

The command you would run in cron would look something like:

pgrep -f celery 2>/dev/null || /path/to/celery_startup

Where /path/to/celery_startup is a simple bash script that contains the commands to be executed (the ones mentioned in the OP).