1

I have a queue processed in separated thread. When I fill jobs from the main thread only the last job is processed multiple time. It works if I block the queue with a join() in between each put() but this don't fill my need. In Python 3:

Q = PriorityQueue()

def processQ():
    while True :
        if not Q.empty():
            job = Q.get(False)
            job[1]()
            Q.task_done()

def fill():
    for i in range(0,100):
        Q.put(((1, Q.qsize()), lambda: print(i)))

def main():
    Thread(target=lambda: processQ()).start()
    fill()

And the output is :

99
99
99
99
etc... 100 times

I've read some things about solving it with multiprocessing but this seems very complicated for the simple behavior I want...

Another thing I don't understand is why I have to include Q.qsize() in the put otherwise a

TypeError: unorderable types: function() < function()

is raised. I had not to do this in Python 2.7

I will be very happy if you could help me

****** EDIT ******

So you CANNOT use lambda function as I did. You HAVE to put function with arguments in the queue with a tuple like this:

for i in range(0,100):
     Q.put(((1, Q.qsize()), (print, i)))

def processQ():
    while True :
        if not Q.empty():
            job = Q.get(False)
            func = job[1][0]  # [0] is the priority, [1:] are the arguments
            args = job[1][1:]
            func(*args)
            Q.task_done()

THE question now is WHY ?

pab4ing
  • 21
  • 1
  • 7
  • Priority queues are not threadsafe. This doesn't solve your problem, but may save you a lot of time. – erip Oct 24 '16 at 11:59
  • 1
    what @erip said, and as to `WHY?` you may want to read some of that: http://docs.python-guide.org/en/latest/writing/gotchas/ – user3012759 Oct 24 '16 at 15:29
  • ok thanks ! the link is very useful ! – pab4ing Oct 29 '16 at 17:36
  • https://docs.python.org/3.5/library/queue.html actually here it's written: "Internally, the module uses locks to temporarily block competing threads; however, it is not designed to handle reentrancy within a thread." meaning threadsafe. – pab4ing Nov 08 '16 at 11:20

0 Answers0