What are the recommended ways to deal with multiprocessing and sleep?

Question

So, I've been playing with Python's multiprocessing module for a few days now, and there's something that I can't understand. Maybe someone can give me a bit of help.

I'm trying to run two methods from the same class in parallel, but apparently there's something that I'm missing:

from multiprocessing import Process
import time


class SomeClass:
    def __init__(self):
        pass

    def meth1(self):
        print(1)
        time.sleep(10)

    def meth2(self):
        print(2)
        time.sleep(5 * 60)


def main():
    while True:
        s = SomeClass()

        p1 = Process(target=s.meth1)  # I want this to run from 10 to 10 seconds
        p1.start()
        p2 = Process(target=s.meth2)  # I want this to run from 5 to 5 minutes
                                      # while the first one still does its own
                                      # job from 10s to 10s
        p2.start()

        p1.join()
        p2.join()


if __name__ == '__main__':
    main()

What I would expect to happen is:

the first method should print 1;
then the second one should print 2 (without waiting 10s - which does happen and seems to work as expected);
then I should see only 1s for the next 4 minutes and 50s (this isn't happening and the program just waits for that time to pass.

What am I missing? Why does the second step work as expected, but the 3rd one doesn't? How can I make it work?

@wp78de feel free to edit the title if you can think of a better one ^^ — Cajuu', Jan 02 '18 at 19:15
`meth1` only prints `1` once. Why would you expect it to keep doing that? — user2357112, Jan 02 '18 at 19:15
@user2357112 because of the `while True`? Should print `1` every 10 seconds :) — Cajuu', Jan 02 '18 at 19:16
Were you expecting the `while` loop to run `meth1` every 10 seconds? That loop waits for both processes to finish. It has to wait for `meth2` to finish before running another `meth1`. — user2357112, Jan 02 '18 at 19:17
Then why does the second method not wait for the first 10 seconds to finish in order to print `2`? — Cajuu', Jan 02 '18 at 19:21
The loop doesn't wait for p1 to finish until after it's already started p2. It waits for both processes to finish before launching a new batch of two processes. — user2357112, Jan 02 '18 at 19:26
So to answer your question: I wasn't expecting the `while` loop to run `meth1` every 10 seconds but instead, I was expecting `meth1` to print `1` and `meth2` to print `2` at the same time (barely), then while `meth2` is sleeping, I was expecting for `meth1` to still do its job and print `1` from 10 to 10 seconds. I thought that's the use of `multiprocessing`. what did I do wrong? — Cajuu', Jan 02 '18 at 19:27
While `meth2` is sleeping, your `main` is waiting at `p2.join()` and `meth1` cannot run again until the `while` loop starts again. — quamrana, Jan 02 '18 at 19:39

score 1 · Answer 1 · answered Jan 02 '18 at 19:41

Think of processes as friends / clones of yours that you can call on the phone, have come in and do something, and then when they are all done they go home, leaving you a note.

The lines:

p1 = Process(target=s.meth1)
p1.start()

call one clone up and have him run s.meth1. He comes over and prints 1 on your screen, waits 10 seconds, then leaves you a note: "all done, no exceptions occurred, I've gone home".

Meanwhile (while your first clone is coming over), the lines:

p2 = Process(target=s.meth2)
p2.start()

call up another clone and have him run s.meth2. He comes over, prints 2 on your screen, waits around for 5 minutes, then leaves you a note: "all done, no exceptions occurred, I've gone home".

While clones #1 and #2 are doing their work, the line:

p1.join()

waits for clone #1 to leave you his note. That happens after 10 seconds. You then go on to:

p2.join()

which waits for clone #2 to leave you his note. That happens after another 4 minutes and 50 seconds. Then you go back around to your while True and start everything over again.

If you want not to wait for clone #2 to finish and leave you his note, don't call p2.join() yet. Eventually, though, you should call p2.join() to make sure that everything went well and that he went home successfully, and isn't lying dead in your living room in a pool of blood, shot by some exception. :-)

Nice ELI5 answer. Though...removing `p2.join()` won't have the desired result either. Removing it will still print `1\n2\n1\n2...` which is not what I want. I'd expect `1\n2\n1\n1\n1\n1\n...2 (after 4 mins and x seconds)... and so on` — Cajuu', Jan 02 '18 at 19:46
"Don't call it *yet*" doesn't mean "don't call it". Moreover, if p2 is still running, you obviously don't want to start it again either. What you might consider doing is waiting for *either one* to finish. See, e.g., [this answer](https://stackoverflow.com/a/26064238/1256452) to a related question. — torek, Jan 02 '18 at 19:49

score 1 · Accepted Answer · answered Jan 02 '18 at 19:51

Its difficult to really know what you want, but this code below does what you describe, but has no convenient way of exiting:

from multiprocessing import Process
import time


class SomeClass:
    def __init__(self):
        pass

    def meth1(self):
        while True:
            print(1)
            time.sleep(10)


    def meth2(self):
        while True:
            print(2)
            time.sleep(5 * 60)


def main():
    s = SomeClass()

    p1 = Process(target=s.meth1)  # I want this to run from 10 to 10 seconds
    p1.start()
    p2 = Process(target=s.meth2)  # I want this to run from 5 to 5 minutes while the first one still does its own job from 10s to 10s
    p2.start()

    p1.join()
    p2.join()



if __name__ == '__main__':
    main()

I have moved the while into each of the methods of SomeClass

This code will never exit, hanging at p1.join()

What are the recommended ways to deal with multiprocessing and sleep?

2 Answers2