Using join() on Processes created using multiprocessing in python

Question

I am using multiprocessing module's Process class to spawn multiple processes, those processes execute some script and then dies.What I wanted, a timeout to be applied on each process, so that a process would die if cant execute in time timeout. I am using join(timeout) on Process objects. Since the join() function doesn;t kill the process, it just blocks the process until it finishes

Now my question : Is there any side-effects of using join() with timeout ..like, would the processes be cleaned automatically, after the main process dies ?? or I have to kill those processes manually ??

I am a newbie to python and its multiprocessing module, please be patient.

My Code, which is creating Processes in a for loop ::

q = Queue()
jobs = [
        Process(
            target=get_current_value,
            args=(q,),
            kwargs=
            {
                'device': device,
                'service_list': service_list,
                'data_source_list': data_source_list
                }
            ) for device in device_list
        ]
for j in jobs:
        j.start()
for k in jobs:
        k.join()

why you don't use signal module ? https://docs.python.org/2/library/signal.html see this question : http://stackoverflow.com/questions/492519/timeout-on-a-python-function-call — Mazdak, Sep 03 '14 at 07:10
The question seems to be dealing with threads...I am using multiprocesses, though — dotslash, Sep 03 '14 at 07:37
What are the child processes actually doing? Are they *consuming* from a `Queue` at all? — dano, Sep 03 '14 at 15:10
@dano : No, they aren't consuming from any queue...they put some result in the queue, rather — dotslash, Sep 05 '14 at 11:02

dano · Answer 1 · 2015-04-19T01:43:44.140

The timeout argument just tells join how long to wait for the Process to exit before giving up. If timeout expires, the Process does not exit; the join call simply unblocks. If you want to end your workers when the timeout expires, you need to do so manually. You can either use terminate, as suggested by wRAR, to uncleanly shut things down, or use some other signaling mechanism to tell the children to shutdown cleanly:

p = Process(target=worker, args=(queue,))
p.start()
p.join(50)
if p.isalive(): # join timed out without the process actually finishing
    #p.terminate() # unclean shutdown

If you don't want to use terminate, the alternative approach is really dependent on what the workers are doing. If they're consuming from a queue, you can use a sentinel:

def worker(queue):
   for item in iter(queue.get, None): # None will break the loop
       # Do normal work

if __name__ == "__main__":
    queue = multiprocessing.Queue()
    p = multiprocessing.Process(target=worker, args=(queue,))
    p.start()
    # Do normal work here

    # Time to shut down
    queue.put(None)

Or you could use an Event, if you're doing some other operation in a loop:

def worker(event):
   while not event.is_set():
       # Do work here

if __name__ == "__main__":
    event= multiprocessing.Event()
    p = multiprocessing.Process(target=worker, args=(event,))
    p.start()
    # Do normal work here

    # Time to shut down
    event.set()

Using terminate could be just fine, though, unless your child processes are using resources that could be corrupted if the process is unexpectedly shut down (like writing to a file or db, or holding a lock). If you're just doing some calculations in the worker, using terminate won't hurt anything.

I am already using the first approach suggested by you...but the join(timeout) doesn't work when I create Processes in a for loop, Its fine for just one Process — dotslash, Sep 04 '14 at 05:47
Should I post the code which creates Processes in a for loop ? — dotslash, Sep 04 '14 at 05:53
@Zira Yes, please edit it into your question. It sounds like you need to put the processes in a list as you start them, and then call `join` on all to the stores after they've all been started. — dano, Sep 04 '14 at 15:27
This is a really good answer! The only thing I would add is that one can also use Python's `signal` package to catch `SIGTERM` and `SIGINT` signals inside the child process in order to still exit gracefully and free resources. Then one can still use `terminate()` to shut down a process. The advantage of this approach over using a `Queue` or an `Event` is that 1) one doesn't need elaborate inter-process communication and 2) it also handles the case of system reboots / shutdowns. — balu, Nov 19 '20 at 17:00
…Finally, if in addition one wraps each process's code entry point in a `try… finally…` block, one can also exit gracefully upon exceptions and free all resources. This way, one pretty much catches all ways a Python process can exit (except, of course, for SIGKILL, which is impossible to catch). — balu, Nov 19 '20 at 17:04

score 0 · Answer 2 · answered Sep 03 '14 at 07:12

0

join() does nothing with the child process. If you really want to terminate worker process in a non-clean manner you should use terminate() (you should understand the consequences). If you want children to be terminated when the main process exits you should set daemon attribute on them.

answered Sep 03 '14 at 07:12

wRAR

25,009
4
84
97

I dont want to go with the non-clean manner..I want my main process to kill those child processes which are taking more time....any clue how to go with this ?? – dotslash Sep 03 '14 at 07:22
1

Either you kill processes or you ask them to terminate cleanly, most likely using whatever channels you already use to pass data to them (like queues), or other IPC ways. You can't cleanly shutdown a process (or a thread, for that matter) from outside. – wRAR Sep 03 '14 at 07:28
I am passing a common queue to each process and process put its result into queue....How could I kill/terminate a process using that queue ??...quite confusing to me – dotslash Sep 03 '14 at 07:40
It seems that join() with timeout, synchronizes all the child process in some manner, unlike join() without any timeout argument – dotslash Sep 03 '14 at 07:48
I'm not sure what do you mean. – wRAR Sep 03 '14 at 10:17
Why is terminate() not clean? The doc says "On Posix OSs the method sends SIGTERM to the child." which means nothing prevents you to catch the signal in the child process and clean everything before exiting. I guess you can do something similar on Windows too (or not if you read the doc for kill().) – Christophe Augier Sep 03 '14 at 16:21
@ChristopheAugier a signal together with a custom signal handler is one of IPC ways... – wRAR Sep 03 '14 at 17:33

Using join() on Processes created using multiprocessing in python

2 Answers2