python multiprocessing does not run functions

Question

I just want to see a simple code implementation of multiprocessing on windows, but it doesn't enter/run functions neither in jupyternotebook or running saved .py

import time
import multiprocessing
s=[1,4]
def subu(remo):
    s[remo-1]=remo*9
    print(f'here{remo}')
    return
if __name__=="__main__":
    p1=multiprocessing.Process(target=subu , args=[1])
    p2=multiprocessing.Process(target=subu , args=[2])
    p1.start()
    p2.start()
    p1.join()
    p2.join()
#     print("2222here")
print(s)
input()

the output by .py is:

[1, 4]
[1, 4]

and the output by jupyternotebook is:

[1,4]

which I hoped to be:

here1
here2
[9,18]

what's wrong with code above? and what about this code:

import concurrent
thread_num=2
s=[1,4]
def subu(remo):
    s[remo-1]=remo*9
    print(f'here{remo}')
    return
with concurrent.futures.ProcessPoolExecutor() as executor:
## or if __name__=="__main__":
##...    with concurrent.futures.ProcessPoolExecutor() as executor:
    results=[executor.submit(subu,i) for i in range(thread_num)]
    for f in concurrent.futures.as_completed(results):
        print(f.result())
input()

doesnot run at all in jupyter pulling error

BrokenProcessPool: A process in the process pool was terminated abruptly while the future was running or pending.

I kinda know I can't expect jupyter to run multiprocessing. but saved.py also can't run it. and it exits without waiting for input()

Running on command line in linux I got the hoped for result. Could not reproduce. — tdelaney, Aug 18 '20 at 01:43
Okay, edit that, not quite the hoped for. `here1\nhere2\n[1,4]`. The [9,18] wouldn't happen because the subprocess memory (either copy-on-write or an independent process) isn't seen by the parent. — tdelaney, Aug 18 '20 at 01:45
@tdelaney so how to apply changes on parent with multiprocessing, and what about 'here1' and 'here2', why they don't happen? — Farhang Amaji, Aug 18 '20 at 01:48
I'm not a regular jupyter user but just poking around saw this [question](https://stackoverflow.com/questions/23641475/multiprocessing-working-in-python-but-not-in-ipython/23641560#23641560) suggesting that the worker be placed in a separate module so that it can be imported by the subprocess. — tdelaney, Aug 18 '20 at 01:50
As for changes back in the parent, you'll need some way to communicate from process back to parent. `multiprocessing.queue()` can be that channel. And `multiprocessing.Pool` can also. Since I don't know the jupyter part, i'm not sure about answering. — tdelaney, Aug 18 '20 at 01:52
@tdelaney I kinda know I can't get multiprocessing with jupyter. and at first I was working with concurrent.future.processpoolexecuter, read reedit of this post agian — Farhang Amaji, Aug 18 '20 at 01:53
I posted a 100% untested answer. Just in case nobody else joins in, its something to try. — tdelaney, Aug 18 '20 at 01:58
I jump into IDEs with the best intentions but always end up with vim and a dozen terminal windows open. — tdelaney, Aug 18 '20 at 02:00

score 1 · Answer 1 · edited Aug 18 '20 at 02:23

1

There are a couple of potential problems. The worker function needs to be importable (at least on Windows) so that it can be found by the subprocess. And since subprocess memory isn't visible to the parent, the results need to be returned. So, putting the worker in a separate module

subumodule.py

def subu(remo):
    remo = remo*9
    print(f'here{remo}')
    return remo

And using a process pool's existing infrastructure to return a worker return value to the parent. You could

import time
import multiprocessing
if __name__=="__main__":
    with multiprocessing.Pool(2) as pool:
        s = list(pool.map(subu, (1,2))) #here
    print(s)
    input()

edited Aug 18 '20 at 02:23

Farhang Amaji

742
11
24

answered Aug 18 '20 at 01:58

tdelaney

73,364
6
83
116

I got ur point in how to treat with s list, and do u mean I can't get "here1' and 'here2' unless I treat them like an output of function? – Farhang Amaji Aug 18 '20 at 02:08
1

I was worried about that. disappointing. Are you on Windows? Just searching "jupyter notebook multiprocessing windows" in that case is my only suggestion. – tdelaney Aug 18 '20 at 02:08
1

You need to get the calculation from `remu()` back to the parent. You could create a `multiprocessing.queue()` then create a `multiprocessing.Process()` and hand the queue down with the call. `Pool` does that part for you. The return value of the worker function is pickled in the process, returned to the parent, unpickled, and then iterated back from the `map` call. `list` just turns that return value iteration into a list. I noticed in your example that initial values in `s` didn't matter. Of course that's just a toy example and you may need to do something different. – tdelaney Aug 18 '20 at 02:14
I didn't get the queue part, do I need `.queue()` for a purpose like, this example of getting handling a separate list. isn't the map enough? – Farhang Amaji Aug 18 '20 at 02:24
1

You wouldn't use a queue if you use map. – tdelaney Aug 18 '20 at 02:25
and what about concurrent.futures.ProcessPoolExecutor? how do I run with that? that seems much easier, and it does remain in console even with input(), and also I tried ``` if __name__=="__main__": with concurrent.futures.ProcessPoolExecutor() as executor:``` and didn't work. – Farhang Amaji Aug 18 '20 at 02:31
another question is that why in my code didn't run `print(f'here{remo}')` from function? – Farhang Amaji Aug 18 '20 at 02:39
I'm not sure. It could be that the process didn't run. It could be that the stdout inherited from the jupyter notebook is directed to nothingness. – tdelaney Aug 18 '20 at 03:01
the output by .py also is: [1, 4] [1, 4] – Farhang Amaji Aug 18 '20 at 03:07

python multiprocessing does not run functions

1 Answers1