Python's multiprocessing manager dict connection error

Question

I am trying to share a dictionary with multiprocessing. The problem is similar to these:

However the proposed solution does not work for me (adding the process.join()). A minimal code example would be:

import multiprocessing as mp
from multiprocessing import Manager


def test(i, multi_cache):
    print(i)
    multi_cache[i] = 'Test'
    print('inside', multi_cache.items())
    


with Manager() as manager:
    multi_cache = manager.dict()

    processes = [mp.Process(target=test, args=(i, multi_cache)) for i in range(3)]
    for p in processes:
        p.start()
    for p in processes:
        p.join()

    print(multi_cache.items())

The output is:

0
1
inside [(0, 'Test'), (1, 'Test')]
2

And afterwards I get this error:

Process Process-8:
Process Process-6:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-3-c78e9bf0d6ec>", line 8, in test
    print('inside', multi_cache.items())
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 749, in _callmethod
    conn = self._tls.connection
  File "<string>", line 2, in items
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 757, in _callmethod
    kind, result = conn.recv()
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
ConnectionResetError: [Errno 104] Connection reset by peer
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "<ipython-input-3-c78e9bf0d6ec>", line 7, in test
    multi_cache[i] = 'Test'
  File "<string>", line 2, in __setitem__
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 753, in _callmethod
    self._connect()
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 741, in _connect
    dispatch(conn, None, 'accept_connection', (name,))
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 79, in dispatch
    kind, result = c.recv()
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer
Traceback (most recent call last):
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 749, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2961, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-c78e9bf0d6ec>", line 21, in <module>
    print(multi_cache.items())
  File "<string>", line 2, in items
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 753, in _callmethod
    self._connect()
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 740, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 487, in Client
    c = SocketClient(address)
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 614, in SocketClient
    s.connect(address)
ConnectionRefusedError: [Errno 111] Connection refused

Weirdly, when only starting one Process the error is slightly different and breaking at the print statement:

Traceback (most recent call last):
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 749, in _callmethod
    conn = self._tls.connection
AttributeError: 'ForkAwareLocal' object has no attribute 'connection'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 2961, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-d7a43009862c>", line 21, in <module>
    print(multi_cache.items())
  File "<string>", line 2, in items
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 753, in _callmethod
    self._connect()
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/managers.py", line 740, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 493, in Client
    answer_challenge(c, authkey)
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 732, in answer_challenge
    message = connection.recv_bytes(256)         # reject large message
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
  File "/home/user/src/anaconda3/envs/pythonenv/lib/python3.6/multiprocessing/connection.py", line 379, in _recv
    chunk = read(handle, remaining)
ConnectionResetError: [Errno 104] Connection reset by peer

I am trying this on ubuntu 16.04 with Python 3.6.10. Any idea how to fix this? Thanks!

Run it from terminal, it fails for me too on Ubuntu with Python 3.8 but only in IPython. Multiprocessing is known to have its quirks in interactive mode. — Darkonaut, Aug 14 '20 at 13:42
This is it! Thank you so much! I've always run it in PyCharm, so I never realized. — hartyrei, Aug 14 '20 at 18:53
You're welcome. More conveniently you can also use "Run" on the script from within PyCharm which emulates a terminal. — Darkonaut, Aug 14 '20 at 19:01
That also worked.. I just wanted to try something out quick on the consol and ended up wasting way more time than just properly writing a script and use run haha.. Thanks so much:) — hartyrei, Aug 14 '20 at 19:18

Python's multiprocessing manager dict connection error

0 Answers0