36

I am using Supervisor (3.0a12) on ubuntu 12.04 to manage php gearman workers. Sometimes the workers get caught in a weird state where they use tons of cpu and ram. While I am figuring this issue out I thought it would be nice to have Supervisor automatically kill and refresh workers occasionally. I looked at http://supervisord.org/configuration.html the configuration documentation and didn't seem to see any options that would allow for this.

Does anyone know if it is possible to have supervisord periodically restart all processes it governs??

dm03514
  • 54,664
  • 18
  • 108
  • 145

3 Answers3

37

You could use crontab to pass commands directly to supervisorctl. For example, the following will restart a process every 20 minutes.

0,20,40 * * * * /path/to/supervisorctl restart [supervisor_process]
scum
  • 3,202
  • 1
  • 29
  • 27
  • This will restart the process but I would suspect it would kill possibly running fine processes too, so could result in data loss with those. (eg. with worker processes not just stateless apps) – tristanbailey Feb 10 '16 at 15:40
  • 6
    Processes die for many reasons: being robust against unexpected restarts is anyway important. So if this strategy would cause data loss, that's a bug, whether or not using cron to achieve this is the right solution. – jma Mar 01 '16 at 11:29
  • I downvoted, what if the supervisorctl fails? what is gonna restart it? the app can stay unloaded forever. Why isn't it included in supervisord? – Alexis Mar 12 '22 at 07:42
31

The superlance package offers a memmon plugin for supervisor. memmon monitors memory usage for programs under supervisor control.

You configure memmon as a supervisor eventlistener:

[eventlistener:memmon]
command=memmon -a 200MB
events=TICK_60

The above configuration sets memmon to restart any program under supervisor control if it exceeds 200MB memory usage. It checks every 60 seconds.

You can configure memmon to monitor specific programs or program groups, setting limits for each.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • If I use Supervisor that uses a process such as "xvfb-run" which starts another process, say xul runner, then does memmon only record the memory that is being user by xvfb, or all the child processes? – CMCDragonkai Oct 29 '13 at 16:49
  • It will only monitor direct child processes, I believe. The manual states: ***memmon** is incapable of monitoring the process status of processes which are not **supervisord** child processes.* The xul runner would not be a child of supervisord, but of xvfb-run. – Martijn Pieters Oct 29 '13 at 16:51
  • A quick scan of [`memmon` codebase](https://github.com/Supervisor/superlance/blob/master/superlance/memmon.py) shows it queries `supervisord` over RPC for the list of programs being managed, and looks directly at pids. You'd have to extend that to look at child processes of those pids too, I guess. – Martijn Pieters Oct 29 '13 at 16:57
  • I think I found a pull request to memmon addressing this? It states it will track the "process tree". https://github.com/Supervisor/superlance/issues/21 – CMCDragonkai Oct 29 '13 at 16:58
  • Yes, that pull request is what you are looking for; it'd take all child processes of a monitored process into account too. – Martijn Pieters Oct 29 '13 at 17:00
  • Just one more question, if I have a supervisor program section using numprocs=X where X is any number greater than 1. Does memmon track the process usage of all processes in that program section, or each process individually? – CMCDragonkai Oct 30 '13 at 07:53
  • 1
    The source code tracks memory per PID, nowhere does it cumulate memory across multiple PIDs. – Martijn Pieters Oct 30 '13 at 09:41
18

There's an easy supervisor-only solution. Make another supervisor process that sleeps for the appropriate amount of time and then restarts the processes you want.

[program:my_program]
process_name = python something.py

[program:periodicrestarter]
command = sh -c "echo 'restart my_program' | supervisorctl;sleep 600"
; restarts all (sleeps first because this restarts itself):
;command = sh -c "sleep 600; echo 'restart all' | supervisorctl"
autorestart = true
Connor
  • 1,226
  • 1
  • 10
  • 18