dumping queue into list/array in python

Question

I am running a number of threads and collecting there result on a queue. I would like to dump it into array or list so that I can do indexing and retrieve those results. Each of the elements in the queue is a array of dimension n. I would like to access those arrays. Would you please let me know, how would i do it?

 def dump_queue(model_queue):
 queue_list = []
 for i in iter(model_queue.get,'STOP'):
         queue_list.append(i)
  return queue_list




aux_model=train_svm(np.array(trainExample),np.array(trainLabel))
model_queue.put(aux_model.coef_)

Thus the arrays are the learned model parameters of svm. model_queue is shared among the threads. I want to access each of the model parameters vectors not each of the entries of a model parameters.

You will have to be a bit more specific. What kind of Queues are these? (python queues, AMQP, STOMP) How is the information represented inside the queue? (json, XML, Yaml, pickled python objects). Also, do you have some code you have tried already and has not worked? How are you planning to index those arrays? — Gabriel Samfira, Jul 18 '13 at 08:35

score 51 · Answer 1 · answered Jul 13 '16 at 11:45

51

You're done with the parallel part and just want to get the results in a list, is that it? Then try:

list(my_queue.queue)

Example:

from queue import Queue

q = Queue()
for i in range(5):
    q.put(i)

l = list(q.queue)
print(l)

Output:

[0, 1, 2, 3, 4]

answered Jul 13 '16 at 11:45

grovina

2,999
19
25

@AkshayLAradhya I've just tested in Python 3.6.5 and it works fine. Are you using the same import? – grovina Jul 25 '18 at 14:02
1

Sorry my bad. I thought this was for `multiprocessing.Queue` – DollarAkshay Jul 25 '18 at 14:11
7

In the interest of full disclosure I think it should noted that the `Queue.queue` attribute isn't documented although it doesn't appear to be private since it has no leading `_` character, which makes its status at least a little bit nebulous in my opinion — in other words it seems like it could simply be an "implementation detail"… – martineau Jan 07 '20 at 20:48
2

Does this leave the items on the queue or removes them? – ed22 Jul 19 '20 at 02:34

score 6 · Answer 2 · answered Aug 22 '14 at 09:42

Not sure whether it's the same problem, but I needed to do the same thing and ended up writing this. I am using threading.Thread and the Queue object in python 2.7. and just wanted to dump out a queue to a list.

def queue_to_list(q):
    """ Dump a Queue to a list """

    # A new list
    l = []

    while q.qsize() > 0:
        l.append(q.get())
    return l

score 1 · Answer 3 · edited Oct 22 '19 at 17:51

Like Gabriel said, you should specifiy your questions.

When you speak of arrays, I think you refer to lists. Python Arrays are used to store numerical values.

However, what you could do to transform your queue to an nested list:# output list:

collectedData = list()
# # # 
# for every thread, call:
threadBuffer = [item for item in q.queue]
collectedData.append(threadBuffer)

This will collect your data in nested lists, which you can easily access, like in this example:

l = list()
l.append([1,2,3,4,5,6,7,9,10])
l.append([6,7,8,9,0,11,22,33])

Now you can access freely: l[1][2] -> 8

Does this help you?

Best regards, Christian

score 1 · Answer 4 · edited May 23 '17 at 11:55

I would go with the following solution:

from collections import deque
q = deque()
for i in range(5):
    q.append([i,i+10])
listed_q = list(q)

Now you can easily access the values by indexing or slicing:

print listed_q[1]        # [1, 11]
print listed_q[1][1]     # 11

Since list() creates a copy of q, you should keep an eye on the dimensionality of your queue. The time needed to do this operation grows the longer your queue is. An alternative approach can be found by using itertools.islice where you first slice the queue before you store the outcome in a list. Check out the following links (performance measures are also given there):

Use slice notation with collections.deque

How to slice a deque? [duplicate]

dumping queue into list/array in python

4 Answers4