4

I have a python application that (in parallel) spawns subprocesses (bash scripts mostly). Some of the scripts may call other scripts. I'm trying to work out the best way to handle termination edge cases for the application, and the subprocesses.

If the application needs to quit, or receives a SIGTERM, then it should terminate (SIGTERM, wait, SIGKILL) all subprocesses and any processes they created. An approach for this would be to start as a new process group and kill the process group as part of the termination (killpg).

If any of the subprocesses takes longer than a specified time, I would like to kill them and and child processes they created. An approach here is to set application as a process group leader so that I may just killpg the group and rely on that to kill any other subprocesses.

The hard bit is that these two solutions conflict with each other, and so I seem to only be able to satisfy one requirement.

So, a final thought is to use tcsetpgrp, but I'm not overly familiar with it. So, something like simulate an interactive terminal. This would mean that killing the application sends a SIGHUP (i think) to all processes, and I can use process groups to manage killing subprocesses that take too long.

Is this a good idea, or are there any other suggestions I'm missing?

Bonus section: If the application is killed via SIGKILL (it is needed occassionally in this application, yes i know SIGKILL should be avoided, etc...), it would be amazing to have the subprocesses killed as well in the same way that bash sends a SIGHUP to its processes when it exits.

tshepang
  • 12,111
  • 21
  • 91
  • 136
Duncan
  • 369
  • 2
  • 10
  • if you can kill an individual subprocess (created with os.setsid) and all its children then to kill all subprocesses spawned by your application you could use a for-loop. I don't see the conflict. – jfs Nov 13 '12 at 11:20
  • Hrm, I forgot the optional bonus section. So that would definitely work for SIGTERM. I would however still like to delve into the world where even a SIGKILL would kill those subprocesses on exit. – Duncan Nov 13 '12 at 16:03
  • Did you find a nice solution for this case? I'm having the same problem in my python program. – ku1ik Jan 16 '13 at 15:13

1 Answers1

0

One possibiltiy to make your scripts self terminating.

Perl has a construct where you can set an alarm.

Good example eslewhere on this site here:

https://stackoverflow.com/questions/3427401/perl-make-script-timeout-after-x-number-of-seconds

Perl, make script timeout after x number of seconds?

There are similar hits searching for python alarm timeout script.

https://stackoverflow.com/questions/1191374/subprocess-with-timeout

Using module 'subprocess' with timeout

This has the side effect (advantage or bug...) that as long as the child process has a shorter timeout than the parent, then the parent may be able to recover gracefully.

It may be better however to limit process by CPU time instead of wall time. In this way a distant descendent doesn't rack up the time for a parent, and if the overall system is slow due to many processes, you don't get a death plague amoung your sub-processes.

You can do this in bash scripts by entering

ulimit -t X

where x is the number of cpu seconds you want. Note however that on most systems, it's a one way street. A process cannot increase it's own limit.

Community
  • 1
  • 1
Sherwood Botsford
  • 1,889
  • 5
  • 20
  • 35