0

I was trying to read and output the shared dictionary element to a file. But I tested that I cannot access the shared dictionary and it will return error.

I wrote the following code but it only works if the dictionary object is not been shared (as I commented).

Could anyone suggest how to access the shared memory dictionary? 

import multiprocessing as mp

sharedi = mp.Manager().dict()
#sharedi = {}
l1 = ['a','b','c','d']
l2 = ['b','b','c','e']

for i in range(0,2):
    name = "abc"+str(i)
    sharedi.update({name:[l1,l2]})

def writ(dirname):
    outfile = open(dirname,'w')
    for i in sharedi:
        for object in sharedi[i][0]:
            outfile.write(object)

print sharedi
p1 = mp.Process(target = writ,args=('d1',))
p2 = mp.Process(target = writ,args=('d2',))

p1.start()
p2.start()
# adding join() as suggeseted in the comment
p1.join()
p2.join()

I'm expecting to write the shared dictionary to files and this is the error message I get:

Process Process-3:
Process Process-2:
Traceback (most recent call last):
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "dict.py", line 14, in writ
    self._target(*self._args, **self._kwargs)
  File "dict.py", line 14, in writ
    for i in sharedi:
    for i in sharedi:
  File "<string>", line 2, in __getitem__
  File "<string>", line 2, in __getitem__
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 755, in _callmethod
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 755, in _callmethod
    self._connect()
    self._connect()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 742, in _connect
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 742, in _connect
    conn = self._Client(self._token.address, authkey=self._authkey)
    conn = self._Client(self._token.address, authkey=self._authkey)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/connection.py", line 169, in Client
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/connection.py", line 169, in Client
    c = SocketClient(address)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/connection.py", line 308, in SocketClient
    c = SocketClient(address)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/connection.py", line 308, in SocketClient
    s.connect(address)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
    s.connect(address)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 228, in meth
    return getattr(self._sock,name)(*args)
error: [Errno 2] No such file or directory
    return getattr(self._sock,name)(*args)
error: [Errno 2] No such file or directory

EDIT:

adding as suggested in the comment, but still get error...

p1.join()
p2.join()

Below is the error message 

Process Process-2:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
Process Process-3:
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "dict.py", line 14, in writ
    self.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "dict.py", line 14, in writ
    for i in sharedi:
  File "<string>", line 2, in __getitem__
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod
    for i in sharedi:
  File "<string>", line 2, in __getitem__
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 774, in _callmethod
    raise convert_to_error(kind, result)
KeyError: 0
    raise convert_to_error(kind, result)
KeyError: 0

UPDATE: I just searched and realized that I cannot (maybe) iterate a shared dictionary object. One way to iterate it is to convert it to normal dict again. But in the program I'm going to write, that would take to much memory(each dict is about 4G). Is there any other way to iterate a shared dict object?

Unsalted Fish
  • 11
  • 4

0 Answers0