4

I'm not able to extract a multiprocessing queue's elements due to AutoProxy error.

Normal way extracting queue's elements without removing is to do - list(q.queue) where q-> queue object. Doesn't seem to work for MP queue.

import multiprocessing as mp

q = mp.Manager().Queue(maxsize=20)

list(q)

TypeError: 'AutoProxy[Queue]' object is not iterable

list(q.queue)

AttributeError: 'AutoProxy[Queue]' object has no attribute 'queue'

list(q.queue.queue)

AttributeError: 'AutoProxy[Queue]' object has no attribute 'queue'

The same happens when I use:

q = mp.Queue(maxsize=20)

I want to be able to extract queue elements without doing a q.get() and want to continue to use a shared queue as it's used by multiple processes.

martineau
  • 119,623
  • 25
  • 170
  • 301
KungFuPanda
  • 55
  • 1
  • 6
  • You try to synchronize your threads with https://docs.python.org/3/library/queue.html – diegoiva Apr 17 '19 at 06:43
  • 1
    I'm using multiprocessing n not multithreading. – KungFuPanda Apr 17 '19 at 06:46
  • Why are you trying to use the manager's Queue? As a proxy it is pointing to a shared queue, It won't accept objects as pool methods. – diegoiva Apr 17 '19 at 06:54
  • Is there any particular reason why you don't use a shared list right away? Yes, it needs explicit locking, but queues simply don't support your requirement from a design point of view. – shmee Apr 17 '19 at 07:02
  • Whether I use manager's queue or just multiprocessing queue, I'm still not able to extract queue contents. I just need the information in the queue elements. – KungFuPanda Apr 17 '19 at 07:02
  • I want to process requests coming in, in a FIFO basis. I ended up using queue for that, I also used locks wherever necessary, n everything worked fine until I wanted to list the queue contents. – KungFuPanda Apr 17 '19 at 07:04
  • You can only access a `multiprocessing.Queue` through its published interface, full stop—so you're going to have to use `get()` method. – martineau Apr 17 '19 at 07:07
  • Another way is to do : new_q= q and do a new_q.get() instead of q.get() . This way, original 'q' contents are unaltered. – KungFuPanda Apr 17 '19 at 07:32
  • 2
    @RojaPatil No. `new_q = q` just adds another name to the Queue object (resp. proxy object) referenced by `q` in your namespace. `new_q.get()` _will_ alter that one underlying object both names refer to. – shmee Apr 17 '19 at 08:01
  • So, how do I copy without referencing it? Is there a way? – KungFuPanda Apr 17 '19 at 08:42
  • 1
    `Manager.Queue` is actually a `queue.Queue` (multithreading-queue, but on a separate server-process) under the hood, not a `multiprocessing.Queue`. Proxy-objects only expose methods, not attributes. You need to subclass `queue.Queue` and expand it with a method returning the attribute, then register this new Queue-class with the manager. You can adapt my solution [here](https://stackoverflow.com/a/53623278/9059420) for your use case (switch `queue.PriorityQueue` for `queue.Queue`). – Darkonaut Apr 17 '19 at 09:39
  • As a workaround, possibly lock the queue, retrieve all items one by one, and then put them all back in. Super inefficient but may work for you for now while you consider @Darkonaut solution above. – MyNameIsCaleb Apr 17 '19 at 13:19
  • Thanks a lot @Darkonaut . I simplified the solution to fit my needs.You are a life saver. Thank you all for your help and support. – KungFuPanda Apr 19 '19 at 04:53
  • @GiraffeMan91 , I had that in my mind initially, but as u say, it's inefficient. I also thought of disguising 'q' as a normal Queue inside a shared MP list, in which case, the q would always be referred as the first element of the list. This way, I can access all the attributes and still have the 'q' shared across. – KungFuPanda Apr 19 '19 at 04:57

1 Answers1

-1

for multiprocessing's Queue..

q = Queue()
q.put(2)
q.put(4)

elements = list()
while q.qsize():
    elements.append(q.get())
print(elements)
jg mtz
  • 19
  • 2
  • What if I just want to read the top element of the queue instead of popping it out by q.get()? – etotientz Sep 12 '20 at 17:55
  • 4
    A q.get() essentially removes the top most element, which I do not want, I want to be able to read the Queue elements w/o modifying the Queue. @Darkonaut's solution at - https://stackoverflow.com/questions/53613583/queue-function-for-multiprocess-priority-queue-in-python-with-syncmanager-class/53623278#53623278 does wonders – KungFuPanda Sep 14 '20 at 06:00