Python multiprocessing, passing an object reference containig a semaphore

Question

I've a scenario like this: I've created an object of the class element containing a semaphore.

import multiprocessing as mpr

class Element(object):
    def __init__(self):
        self.sem = mpr.Semaphore()
        self.xyz = 33

def fun( ch ):
    a = ch.recv()
    print( a[0] )
    print( a[1].xyz )
    a[1].xyz = 99
    print( a[1].xyz )


el = Element()

( pa , ch ) = mpr.Pipe()
proc = mpr.Process(target=fun , args=( ch, ) )

proc.start()
pa.send( [ "Hallo" , el ])

print( el.xyz )

proc.join()

This code return this error:

  File "/usr/lib/python2.7/multiprocessing/forking.py", line 51, in assert_spawning
    ' through inheritance' % type(self).__name__
RuntimeError: Semaphore objects should only be shared between processes through inheritance

But if I remove the semaphore from the declaration of Element the code works, but the value assigned to a[1].xyz will be lost.

Now I need to synchronizes a big collection of object via semphore and multiprocessing. So there's some method for setting a semaphore in every object and passing only the reference to the main object?

import multiprocessing as mpr

class Element(object):
    def __init__(self):
        self.xyz = 33

def fun( ch ):
    a = ch.recv()
    print( a[0] )
    print( a[1].xyz )
    a[1].xyz = 99
    print( a[1].xyz )


el = Element()

( pa , ch ) = mpr.Pipe()
proc = mpr.Process(target=fun , args=( ch, ) )

proc.start()
pa.send( [ "Hallo" , el ])

print( el.xyz )

proc.join()

The second version dot't produce any error, but the value assigned to a[1].xyz = 99 will be lost in the main process.

Your error is clear it tells that the semaphore should be an attribute of the Process and you can't send it to him. So basically you need to inherit from Process and define a `run` etc... But it's way more obscure what you're trying to do. — Rik Poggi, Oct 19 '12 at 12:40
Maybe you could allocate a large array of semaphores at the start and pass just keep a sem_index inside the Element object? — cdleonard, Oct 19 '12 at 12:41
Passing an array reproduce the same problem in a different way, the problem is with the semaphore, not in the container. — Giggi, Oct 19 '12 at 12:45

Bakuriu · Accepted Answer · 2012-10-19T17:03:35.680

I don't think you understood how the multiprocessing module works.

When you send something through the pipe, it gets pickled and then unpickled in the subprocess. This means that the subprocess actually has a copy of the original object! That's why the change is "lost". Adding a semaphore wont change anything.

If you want an object in shared memory you should use multiprocessing.Value, even though this does not handle arbitrary types. Probably multiprocessing.Manager is what you are looking for.

An other way would be to send a response to the main process providing the modified object.

Python multiprocessing, passing an object reference containig a semaphore

1 Answers1

Linked