How to obtain the result from multi-processing

Question

I want to use Python multiprocessing to accelerate my program, and my code work:

from multiprocessing import Process

class MyProcess(Process):
    def __init__(self,name, array):
        super(MyProcess,self).__init__()
        self.name = name
        self.array = array

    def run(self):

        s = 0
        for a in self.array:
            s += a
        self.s = s


if __name__ == '__main__':
    process_list = []
    for i in range(5):
        p = MyProcess(str(i), [1, 2, 3, 4, 5])
        p.start()
        process_list.append(p)

    for p in process_list:
        p.join()

    # for p in process_list:
    #     print(p.s)

In this example code, I want to calculate the sum of the input array. How can I obtain the calculated result?

print(p.s) reports bug: MyProcess object has no attribute 's'.

Any suggestion is apprecated~~~

The reason why it does not work is that `__init__` runs inside the main process, but `run` runs inside the subprocess - there are different copies of the `MyProcess` instance in each. When you try to print `p.s`, you are in the main process. You will see this if you print out the value of `os.getpid()` inside each of these. There are some possible solutions at https://stackoverflow.com/questions/10415028/how-can-i-recover-the-return-value-of-a-function-passed-to-multiprocessing-proce - do they help you? — alani, Aug 23 '22 at 12:10
Consider use `ProcessPoolExecutor` from `concurrent.futures`. It provide `result()` method to get the return value from executed process. https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor — HALF9000, Aug 23 '22 at 12:14
Does this answer your question? [How can I recover the return value of a function passed to multiprocessing.Process?](https://stackoverflow.com/questions/10415028/how-can-i-recover-the-return-value-of-a-function-passed-to-multiprocessing-proce) — Timus, Aug 23 '22 at 15:25
maybe you should use `numpy` for calculations because it uses code created in `C/C++` and it can run faster (even without multiprocessing) — furas, Aug 23 '22 at 15:36

score 1 · Answer 1 · answered Aug 23 '22 at 12:17

1

You can use ProcessPoolExecutor from concurrent.futures and collect the result:

from concurrent.futures import ProcessPoolExecutor

input_list = [1, 2, 3, 4, 5]


def calculate_sum(a: list):
    return sum(a)


if __name__ == "__main__":
    future_obj = ProcessPoolExecutor().submit(calculate_sum, input_list)
    print(future_obj.result())

answered Aug 23 '22 at 12:17

Dan Constantinescu

1,426
1
7
11

Can `ProcessPoolExecutor` be used for `MyProcess(Process)`? Or it can only be used to `submit` a `calculate_sum` function? – yingzi Aug 23 '22 at 14:19
1

@yingzi when you use `ProcessPoolExecutor` then you don't have to create own `MyProcess` (`Pool` creates own `Processes` for this) and use normal function. It will be simpler. – furas Aug 23 '22 at 15:34

score 0 · Answer 2 · answered Aug 24 '22 at 01:56

Thanks every helpful answer. And this post help me a lot. However, the post provide many answers to collect answer for function subprocess rather than class subprocess.

I have tried a solution to collect answer for class subprocess, and the code is:

import multiprocessing
from multiprocessing import Process
import numpy as np

class MyProcess(Process):
    def __init__(self,name, array):
        super(MyProcess,self).__init__()
        self.name = name
        self.array = array
        recv_end, send_end = multiprocessing.Pipe(False)
        self.recv = recv_end
        self.send = send_end

    def run(self):

        s = 0
        for a in self.array:
            s += a
        self.send.send(s)

    def getResult(self):
        return self.recv.recv()


if __name__ == '__main__':
    process_list = []
    for i in range(5):
        a = np.random.random(10)
        print(i, ' correct result: ', a.sum())
        p = MyProcess(str(i), a)
        p.start()
        process_list.append(p)

    for p in process_list:
        p.join()

    for p in process_list:
        print(p.name, ' subprocess result: ', p.getResult())

How to obtain the result from multi-processing

2 Answers2