2

I put an object containing a multiprocessing.Queue in a multiprocessing.Queue and I get the following error:

File "/usr/local/lib/python3.4/multiprocessing/queues.py", line 242, in _feed
  obj = ForkingPickler.dumps(obj)
File "/usr/local/lib/python3.4/multiprocessing/reduction.py", line 51, in dumps
  cls(buf, protocol).dump(obj)
File "/usr/local/lib/python3.4/multiprocessing/queues.py", line 57, in __getstate__
  context.assert_spawning(self)
File "/usr/local/lib/python3.4/multiprocessing/context.py", line 347, in assert_spawning
  ' through inheritance' % type(obj).__name__
RuntimeError: Queue objects should only be shared between processes through inheritance

I understand that the error is caused when the multiprocessing.Queue attempts to pickle the multiprocessing.Queue. I've tried the solution of using a multiprocessing.Manager to create the queue as suggested in Sharing a queue among several processes and it works in the sense that I can add the managed queue to another multiprocessing.Queue. However, I run into this issue.

Is there a way to fix the pickling issue in my object so the object can contain a multiprocessing.Queue() instead of a multiprocessing.Manager().Queue()?

Community
  • 1
  • 1
harumph
  • 195
  • 1
  • 11
  • 1
    I think you're better off trying to fix the issue with `multiprocessing.Manager().Queue`, rather than trying to make `multiprocessing.Queue` picklable. `multiprocessing.Queue` (and `multiprocessing.Lock`, which are used to implement `Queue`) are both purposely made to be not picklable except when being passed to a spawning `Process` object. Trying to hack around that is likely to result in code that doesn't work properly on all platforms. – dano Jan 14 '15 at 16:17

0 Answers0