Using the multiprocessing module

Question

I'm trying to use the multiprocessing module in python 2.6, but apparently there is something I do not understand. I would expect the class below to add up the numbers sent to it by add() and return the sum in the get_result() method. The code below prints "0", I'd like it to print "2". What have I missed?

import multiprocessing

class AdderProcess(multiprocessing.Process):

    def __init__(self):
        multiprocessing.Process.__init__(self)
        self.sum = 0
        self.queue = multiprocessing.JoinableQueue(5)
        self.daemon = True
        self.start()

    def run(self):
        while True:
            number = self.queue.get()
            self.sum += number
            self.queue.task_done()

    def add(self, number):
        self.queue.put(number)

    def get_result(self):
        self.queue.join()
        return self.sum


p = AdderProcess()
p.add(1)
p.add(1)
print p.get_result()

PS. This problem has been solved. Thanks for the answers! Just to make it easier for any readers, here's the complete working version:

import multiprocessing

class AdderProcess(multiprocessing.Process):

    def __init__(self):
        multiprocessing.Process.__init__(self)
        self.sum = multiprocessing.Value('d', 0.0)
        self.queue = multiprocessing.JoinableQueue(5)
        self.daemon = True
        self.start()

    def run(self):
        while True:
            number = self.queue.get()
            self.sum.value += number
            self.queue.task_done()

    def add(self, number):
        self.queue.put(number)

    def get_result(self):
        self.queue.join()
        return self.sum.value

p = AdderProcess()
p.add(1)
p.add(1)
print p.get_result()

unutbu · Accepted Answer · 2011-10-15T21:21:59.163

Change self.sum = 0 to self.sum = multiprocessing.Value('d', 0.0), and use self.sum.value to access or change the value.

class AdderProcess(multiprocessing.Process):    
    def __init__(self):
        ...
        self.sum = multiprocessing.Value('d', 0.0) 
        ...
    def run(self):
        while True:
            number = self.queue.get()
            self.sum.value += number    # <-- use self.sum.value
            self.queue.task_done()
    def get_result(self):
        self.queue.join()
        return self.sum.value           # <-- use self.sum.value

The problem is this: Once you call self.start() in __init__, the main process forks off a child process. All values are copied. Now there are two versions of p. In the main process, p.sum is 0. In the child process, the run method is called and p.sum is augmented to 2. But when the main process calls p.get_result(), its version of p still has p.sum equal to 0. So 0 is printed.

When you want to share a float value between processes, you need to use a sharing mechanism, such as mp.Value.

See "Sharing state between processes" for more options on how to share values.

Assuming that you mean that I should replace my initializer with your code, that code causes a TypeError exception when executed. Did it work for you? Can you elaborate on what you are doing here? — Mats Ekberg, Oct 15 '11 at 20:57
Sorry, I did run the code, I just forgot to include all my changes. You need to also change `self.sum` to `self.sum.value` to access or change the value. — unutbu, Oct 15 '11 at 21:00
Ok, I think I got it now. There is no RPC magic that makes the method calls go to the other process, the only shared information is the one explicitly shared with the data types provided for that purpose. Makes sense I guess. Thanks! — Mats Ekberg, Oct 15 '11 at 21:19

score 1 · Answer 2 · answered Oct 15 '11 at 20:57

self.sum is 2... in that process:

def run(self):
    while True:
        number = self.queue.get()
        print "got %s from queue" % number
        print "Before adding - self.sum = %d" % self.sum
        self.sum += number
        print "After adding - self.sum = %d" % self.sum
        self.queue.task_done()

[ 13:56 jon@host ~ ]$ ./mp.py
got 1 from queue
Before adding - self.sum = 0
After adding - self.sum = 1
got 1 from queue
Before adding - self.sum = 1
After adding - self.sum = 2

See multiprocessing 16.3.1.4. - Sharing state between processes on how to get self.sum to be the same in different processes.

Using the multiprocessing module

2 Answers2

Linked