22

I need to read strings written by multiprocessing.Process instances from the main process. I already use Managers and queues to pass arguments to processes, so using the Managers seems obvious, but Managers do not support strings:

A manager returned by Manager() will support types list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Queue, Value and Array.

How do I share state represented by a string using Managers from the multiprocessing module?

Bengt
  • 14,011
  • 7
  • 48
  • 66

2 Answers2

31

multiprocessing's Managers can hold Values which in turn can hold instances of the type c_char_p from the ctypes module:

>>> import multiprocessing
>>> import ctypes
>>> v = multiprocessing.Value('c', "Hello, World!")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.7/multiprocessing/__init__.py", line 253, in Value
    return Value(typecode_or_type, *args, **kwds)
  File "/usr/lib/python2.7/multiprocessing/sharedctypes.py", line 99, in Value
    obj = RawValue(typecode_or_type, *args)
  File "/usr/lib/python2.7/multiprocessing/sharedctypes.py", line 73, in RawValue
    obj.__init__(*args)
TypeError: one character string expected
>>> cstring = multiprocessing.Value(ctypes.c_char_p, "Hello, World!")
>>> cstring
<Synchronized wrapper for c_char_p(166841564)>
>>> cstring.value
'Hello, World!'

For Python 3, use c_wchar_p instead of c_char_p

See also: Post with the original solution that I had a hard time finding.

So a Manager can be used to share a string beneath multiple processes in Python like this:

>>> from multiprocessing import Process, Manager, Value
>>> from ctypes import c_char_p
>>> 
>>> def greet(string):
>>>     string.value = string.value + ", World!"
>>> 
>>> if __name__ == '__main__':
>>>     manager = Manager()
>>>     string = manager.Value(c_char_p, "Hello")
>>>     process = Process(target=greet, args=(string,))
>>>     process.start()
>>>     process.join()    
>>>     print string.value
'Hello, World!'
mherzog
  • 1,085
  • 1
  • 12
  • 24
Bengt
  • 14,011
  • 7
  • 48
  • 66
  • 2
    I was wondering why this works (sharing a pointer), and from the sources I learned that `managers.Value` does not actually use the `typecode` argument at all. The string is not converted to `c_char_p`. `multiprocessing.Value` works very differently. – Janne Karila Jan 23 '14 at 12:03
  • 1
    I tried `s = multiprocessing.Value(c_char_p, "old string")` but `s.value = "new string"` often lost the pointer. I also tried another solution of `Array('c', fixed_length)` which is more stable. – John Lin Nov 02 '18 at 09:45
  • In case if you try to get rid of `Manager` it won't work since `c_wchar_p` is only a pointer to the data, so when another process accesses it, it goes directly to `segmentation fault`. More explanation can be found here: https://stackoverflow.com/a/12343115/3300539. And yes, as it was mentioned in the previous comment, `Array('c', fixed_length)` works for me as well. – VladVin Jun 07 '21 at 10:01
  • This did not work for me. I get this exception: ```bytes or integer address expected instead of str instance ``` – Ryan Glenn Jul 31 '22 at 00:21
10

Simply put the string in a dict:

d = manager.dict()
d['state'] = 'xyz'

As strings themselves are immutable, sharing one directly would not be as useful.

Janne Karila
  • 24,266
  • 6
  • 53
  • 94
  • 3
    `manager.dict()` is slower because it returns a proxy, and proxies always copy data from one thread to another, while `multiprocessing.Value()` is faster because it returns an actual shared memory region, which is just directly accessed by each process. – Evandro Coan Aug 31 '19 at 04:19