Forking, daemonizing and monitoring a python script

Question

I have a python script sc.py on my debian server.

I would like :

To run it as a daemon and keep it running indefinitely
To run it N times (as sub processes, children, forks) in //
To monitor the whole processes (in order to restart any dying process)

Do you know software that would enable me to do so?

I don't know if I have to look for a solution on the python side (any python module or configuration) or if there is a debian package somewhere which does that job?

score 2 · Accepted Answer · answered May 15 '17 at 08:27

2

You can use supervisord for this.

It daemonizes Python processes for you, and also handles subprocesses.

answered May 15 '17 at 08:27

zmbq

38,013
14
101
171

Thanks for helping. Do you know if there is the same kind of tool that works with any kind of script (such as bash) ? – Antho May 15 '17 at 12:25
1

supervisord should work with anything you can execute, not just Python. It is itself written in Python though. – Florian Sesser May 15 '17 at 18:12
Yes, that's what I figured out and I went for this solution that perfectly matches my needs. Thanks again. ;) – Antho May 16 '17 at 20:57

score 1 · Answer 2 · edited May 23 '17 at 10:31

1

If this is a one-time task (that is, you are not creating a software product) I would quick-and-dirty use a combination of shell scripting and a terminal multiplexer like screen. For restarting processes that died (that is, they emitted an exit code other than 0), just use the shell.

Start your processes for example like this:

for i in n/*; do
    screen -d -m -L -S $i -t $i until python sc.py $i; do echo "Crashed with exit code $?.  Respawning.." >&2 ; sleep 1 ; done
done

This would

Create a new screen session for every file in the subdirectory n/ running your script,
Restart your python script unless it exited successfully (using Bash's until)
Set the title (-t) and session name (-S) to the input file name,
And turn on logging of all the output for later inspection if something went wrong (-L).

You can then use normal screen commands like screen -list to list all running tasks and screen -r <session name> to view the running session output.

edited May 23 '17 at 10:31

Community

1
1

answered May 15 '17 at 10:26

Florian Sesser

5,972
1
25
26

Thanks for the consideration and the time spent. I need my script run as a service from boot to shutdown, and during its whole life, it will process requests as they arrive. Do you think screen is the solution? – Antho May 15 '17 at 12:26
@Antho Welcome! No, I advise against screen then. I read some people actually use setups like this in production *cough cough*, but I would recommend against running this non-interactively. zmbq's answer (supervisord) is better suited in that case. – Florian Sesser May 15 '17 at 18:10
So we agreed... I did forget about screen, thanks for the reminder! – Antho May 16 '17 at 21:00

score 1 · Answer 3 · answered May 17 '17 at 07:58

1

Another process manager I stumbled upon is circus.

It looks more adventurous, if that's your thing ;)

And the web interface is richer than supervisord's: See some nice screenshots.

answered May 17 '17 at 07:58

Florian Sesser

5,972
1
25
26

1

Looks a bit more complex but it may help someday if my needs change. Thanks for the followup :) – Antho May 18 '17 at 10:49

Forking, daemonizing and monitoring a python script

3 Answers3