77

Im looking for something better than sudo restart projectname every time I issue a git pull origin master, which pulls down my latest changes to a Django project. This restart command, I believe, is related to Upstart, which I use to start/top my Gunicorn server process.

This restart causes a brief outage. Users hitting the web server (nginx) will get a 500, because Gunicorn is still restarting. In fact, it seems to restart instantly, but it takes a few seconds for pages to load.

Any ideas on how to make this seamless? Ideally, I'd like to issue my git pull and Gunicorn reloads automatically.

Ben Keating
  • 8,206
  • 9
  • 37
  • 37

12 Answers12

109

You can tell Gunicorn to reload gracefully using the HUP signal like so:

kill -HUP <pid>

(see the FAQ for details)

I use Supervisor to control my Gunicorn server, which allows me to use this (slightly hacky) way of reloading Gunicorn after a deploy:

supervisorctl status gunicorn | sed "s/.*[pid ]\([0-9]\+\)\,.*/\1/" | xargs kill -HUP

You could obviously achieve something similar with pidof, or ps.

This is actually run from a Fabric script, so I don't even have to logon to the server at all.

Martey
  • 1,631
  • 2
  • 13
  • 23
Rob Golding
  • 3,502
  • 5
  • 26
  • 28
  • 2
    with supervisor you can still ask gunicorn to write its pid to a file if you want with [pidfile](http://gunicorn.org/configure.html#pidfile) but BE CAREFUL to put it somewhere where the user setup by supervisor to run gunicorn will have the right to write! – Stefano May 25 '12 at 11:15
  • 1
    Thanks - that's a neater way of doing it! – Rob Golding May 25 '12 at 14:20
  • 4
    actually, the [pidfile setting](http://supervisord.org/configuration.html?highlight=pidfile) is only supported for the supervisord process itself. And instead of all that sed'ing, why not use `supervisorctl pid gunicorn | xargs ...` ?! – captnswing Jan 03 '13 at 15:28
  • pid = Get the PID of supervisord not of the subprocess – armonge Feb 05 '13 at 16:21
  • 4
    From `supervisorctl help pid`: `pid Get the PID of a single child process by name.` (Which a quick test verifies). Without the name it's `supervisorctl`'s PID. – Julian Mar 13 '13 at 16:30
  • @Julian I disagree. I've just tested it `$ sudo supervisorctl pid program1`, `$ sudo supervisorctl pid program2`, `$ sudo supervisorctl pid program3`, and I get every time the same number. Perhaps it has changed in newer versions (I'm using 3.0a8) – glarrain Jun 21 '13 at 20:40
  • At least on supervisor 3.0, `sudo supervisorctl pid gunicorn` does indeed return the main gunicorn (not supervisord) pid. So then you can run `sudo kill -HUP \`sudo supervisorctl pid gunicorn\`` to gracefully restart Gunicorn. However note that this doesn't necessarily mean zero issues during deployment, because the old workers stop taking requests and your new workers Python app takes some time to boot up, you'll get hanging requests while the new workers get started. – philfreo Sep 25 '13 at 17:18
  • 1
    How does -HUP differ from `supervisorctl reload`? – Scott Stafford Apr 02 '14 at 14:43
  • 3
    @ScottStafford, `supervisorctl reload` asks the supervisor daemon to reload its config. Perhaps you meant `supervisorctl restart`? This is just shorthand for stop followed by start, so the gunicorn master and all workers are clobbered. Using `SIGHUP` leaves the gunicorn master running; new workers are created, old ones gracefully killed. No requests are dropped. Much cleaner. – Ben Graham Apr 16 '14 at 06:08
  • just an easier version when using supervisor: `supervisorctl pid NAME | xargs kill -HUP` – gru Jan 21 '15 at 14:46
  • 2
    `supervisorctl signal HUP NAME` should also do the trick. http://supervisord.org/running.html#signals – Ben Oct 14 '16 at 20:30
37

For those not using supervisord: what Rob said, it works with ps as well,

ps aux |grep gunicorn |grep projectname | awk '{ print $2 }' |xargs kill -HUP
littlegreen
  • 7,290
  • 9
  • 45
  • 51
  • 1
    What would be the difference between these: $ ps -A | grep gunicorn | awk '{print $1}' | xargs \\\ $ "15135 15189 15193 15196" \\\ $ ps aux | grep gunicorn | grep staging | awk '{print $2}' \\\ $ "15193 15196" \\\ obviously, your version is not picking up all of the gunicorn processes. Only resetting your version worked fine. But what are these other two processes? – Michael Bruce Aug 30 '16 at 19:25
21

For a graceful reload, you should instead use Upstart's reload command, e.g.:

sudo reload jobname

According to the initctl (Upstart) manpage, reload will send a HUP signal to the process:

reload JOB [KEY=VALUE]...

       Sends the SIGHUP signal to running process of the named JOB instance.

...which for Gunicorn will trigger a graceful restart (see FAQ).

johntellsall
  • 14,394
  • 4
  • 46
  • 40
Gary
  • 3,425
  • 25
  • 18
  • 3
    noob question: and is `jobname` something I get from the process itself or an option I passed into gunicorn? – TheGrimmScientist Feb 14 '16 at 19:53
  • 2
    @TheGrimmScientist, `jobname` is the name of your upstart job/config. E.g. For the upstart job at `/etc/init/foo.conf`, `foo` would be the jobname. – Gary Jun 01 '16 at 04:37
  • @Gary Is there any example available? – Dewsworld May 29 '17 at 12:33
  • 1
    @Dewsworld, an example upstart config? Many examples can be found, e.g. https://stackoverflow.com/questions/17747605/daemon-vs-upstart-for-python-script For a full reference, see: http://upstart.ubuntu.com/cookbook/ – Gary Jun 14 '17 at 21:42
  • In the context of this question, try `sudo reload /etc/init/gunicorn.conf`. This assumes you've got `gunicorn.conf` at `/etc/init/`. – Hassan Baig Oct 06 '17 at 10:55
16

Systemd, gunicorn & Ubuntu

Here is the one-liner, if you are running your gunicorn service with systemd.

systemctl status gunicorn |  sed -n 's/.*Main PID: \(.*\)$/\1/g p' | cut -f1 -d' ' | xargs kill -HUP

Details step by step

Since the gunicorn docs tell that the correct way to gracefully reload the workers is by using kill -HUP <Main PID>, where <Main PID> is the process id of the master process, we extract the master PID using systemctl, and run kill -HUP <Main PID>.

1) Get info about the process from systemd using the name of the service

systemctl status gunicorn

where gunicorn is the name of the service, located at /etc/systemd/system/.

Example output:

ubuntu@ip-10-4-12-247:~$ systemctl status gunicorn
● gunicorn.service - Gunicorn server for yourproject.com
   Loaded: loaded (/etc/systemd/system/gunicorn.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2017-11-04 19:16:24 UTC; 1h 15min ago
 Main PID: 10673 (gunicorn)
   CGroup: /system.slice/gunicorn.service
           ├─10673 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application
           ├─11069 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application
           ├─11070 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application
           └─11071 /home/ubuntu/site/venv/bin/python3 /home/ubuntu/site/venv/bin/gunicorn --workers 3 --bind unix:/tmp/yourproject.socket config.wsgi:application

Nov 04 20:27:04 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:27:04 +0000] [11047] [INFO] Booting worker with pid: 11047
Nov 04 20:27:04 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:27:04 +0000] [11048] [INFO] Booting worker with pid: 11048
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [10673] [INFO] Handling signal: hup
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [10673] [INFO] Hang up: Master
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11046] [INFO] Worker exiting (pid: 11046)
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11047] [INFO] Worker exiting (pid: 11047)
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11048] [INFO] Worker exiting (pid: 11048)
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11069] [INFO] Booting worker with pid: 11069
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11070] [INFO] Booting worker with pid: 11070
Nov 04 20:32:16 ip-10-4-12-247 gunicorn[10673]: [2017-11-04 20:32:16 +0000] [11071] [INFO] Booting worker with pid: 11071

2) Get the process ID (PID) of the main gunicorn process

The sed command works like follows: sed 's/<search this>/<replace with this>/g'

  • s means for the substitute command, and g means that search the whole input globally.
  • The -n flag tells sed not to print every line (or actually, not to print anything.)
  • The p at the end tells sed to print the matched line.
  • We search for .*Main PID: \(.*\)$, which is a regular expression pattern, which has following parts: .* matches any character (.) zero or more times (*). Then we search for Main PID: followed by any characters, repeated zero or more times (.*). To capture all characters after the Main PID:-text, we enclose the .* into parenthesis, which are escaped with backslashes: \(.*\). $ indicates line end.
  • The "replace with this" part of the sed command is just \1, which means the first captured set of characters.

Example output:

ubuntu@ip-10-4-12-247:~$ systemctl status gunicorn |  sed -n 's/.*Main PID: \(.*\)$/\1/g p'
10673 (gunicorn)

3) Get rid of the extra characters

Pipe the output to cut. The cut -f1 -d' ' means, that

  • The string is space delimited: Here -d determines the delimiter, which is the characted just after the -d. Since the delimiter is space, we enclose that in quotes.
  • -f means just that cutting is made using the delimiter (and not by bytes), and -f1 means that we want to take out the first element of the list.

Example output:

ubuntu@ip-10-4-12-247:~$ systemctl status gunicorn |  sed -n 's/.*Main PID: \(.*\)$/\1/g p' | cut -f1 -d' '
10673

4) Use the Main PID

Piping to xargs means just running the command with arguments from the pipe on the left hand side. Since we are piping just the Main PID to xargs,

 systemctl status gunicorn-django |  sed -n 's/.*Main PID: \(.*\)$/\1/g p' | cut -f1 -d' ' | xargs kill -HUP

is basically just the same thing as 

echo <Main PID > | xargs kill -HUP

which translates into

kill -HUP <Main PID >

Edit

A little more robust solution would be to use cut -f1 -d$'\n' or grep -m1 "" in front of cut -f1 -d' ', to pick just the first line of the match. I can't figure out any circumstances, where there would be two matches for the Main PID:, though.

Niko Föhr
  • 28,336
  • 10
  • 93
  • 96
7

Maybe not a direct answer to the question, but for those who are just looking for a way to restart gunicorn web server, you can use killall gunicorn and then execute a command to start gunicorn again. For example: 

killall gunicorn
gunicorn --bind 0.0.0.0:80 --reload app:app

Note: killall gunicorn will terminate all gunicorn processes immediately so make sure you understand what you are doing.

Mykhailo Seniutovych
  • 3,527
  • 4
  • 28
  • 50
  • 2
    While that will work, it’s likely bad practice. Reloading gunicorn gracefully will allow active connections to finish as they should. Just killing things will break those connections and if something is left open And gunicorn ain’t there to say close it, you could get into some lame issues. However, If your site/project/constraints are simple enough this is fine to me, sure. – Ben Keating Oct 25 '18 at 15:31
6

We run Gunicorn under Supervisor, but this is the simplest, cleanest way we've found to gracefully reload Gunicorn when it gets confused:

sudo pkill -HUP -f gunicorn.*master
johntellsall
  • 14,394
  • 4
  • 46
  • 40
3
sudo systemctl restart gunicorn
Muhammad Dyas Yaskur
  • 6,914
  • 10
  • 48
  • 73
Odiljon Djamalov
  • 761
  • 8
  • 11
  • Unfortunately this will also give users hitting the web server (nginx) a 500, because Gunicorn is still restarting – curtisp Oct 16 '20 at 14:58
1

If you are running gunicorn on a port rather than a socket, you can find the process id (pid) of gunicorn using fuser command. Then force gunicorn to reload the code by sending a HUP signal.

The command fuser 8000/tcp will list the process ids of all processes using tcp port 8000.

fuser -k 8000/tcp will kill those processes gracelessly which is not recommended.

  • fuser -k -HUP 8000/tcp will force gunicorn using tcp port 8000 to reload the code by sending HUP signal.
Mohammed Shareef C
  • 3,829
  • 25
  • 35
1

When using systemd

calling:

systemctl restart gunicorn

resulted in 502 bad gateway errors for us. Digging into the nginx logs we saw that the socket file was not found. Calling

systemctl restart gunicorn.socket

after restarting gunicorn, restarted the socket and brought the server back online.

user2682863
  • 3,097
  • 1
  • 24
  • 38
0

Restart gunicorn service using cmd
systemctl restart gunicorn

else restart gunicorn service and create the sock file again.

Avi
  • 1,424
  • 1
  • 11
  • 32
dhanya a
  • 31
  • 1
0

You can start gunicorn with the code:

gunicorn  --pid /var/run/gunicorn.pid --bind 0.0.0.0:80 --reload myproject.asgi.application

And graceful restart with the code:

cat /var/run/gunicorn.pid | xargs kill -HUP
WoodenRobot
  • 1
  • 1
-1

This can help when gunicorn restart(in a dev/test env) takes a lot of time to restart.

The below command kills the workers.

$ sudo kill -9 `sudo lsof -n -i | grep gunicorn | awk '{print $2}'`

$ sudo service gunicorn restart
SuperNova
  • 25,512
  • 7
  • 93
  • 64