2

If I run a thread with the following function as the worker,

q = queue.Queue()

def worker():
    while True:
        t = {}
        for i in range(3):
            t['a'] = i
            q.put(t)

the queue is populated with dictionaries that are all the same, i.e., {'a': 2} instead of the sequence {'a': 0}, {'a': 1}, {'a': 2}. I assume this is because the put() method runs after the for loop has finished and the last value of i was 2. Am I interpreting that right?

Now, if I move the instantiation of the dictionary inside the for loop,

def worker():
    while True:
        for i in range(3):
            t = {'a': i}
            q.put(t)

the queue is populated with the desired sequence. My interpretation is that in the first instance, I create a dictionary object in memory, then begin a for loop and reassign its value 3 times but the put() calls happen after the loop has finished. In the second instance, I create a new dictionary object every iteration of the for loop and so when the put() calls occur after the loop, they access 3 distinct instances of the dictionary with their own key-value pairs.

Can anyone shed some light on what's happening behind the curtain here?

wrkyle
  • 529
  • 1
  • 13
  • 36
  • No, Queue.put is not asynchronous. You are simply adding *the same dictionary* over and over again. You did not create copies. See the duplicate. – Martijn Pieters Feb 11 '18 at 00:27

2 Answers2

3

Am I interpreting that right?

You observe such behavior because you’re modifying the same object all the time

Lets put aside queues / threads and run a simplified equivalent of your code with some prints to understand what’s happening

t = {}
l = []
for i in range(3):
    t['a'] = i
    l.append(t)
print(l)
t['a'] = 20
print(l)
print(map(id, l))

[{'a': 2}, {'a': 2}, {'a': 2}]
[{'a': 20}, {'a': 20}, {'a': 20}]
# they are all the same!
[4474861840, 4474861840, 4474861840]

So it has nothing to do we threads/queues - you’re just adding the same object 3 times.

Now, if I move the instantiation of the dictionary inside the for loop

In this case you create a new object every time like in the following code:

l = []
for i in range(3):
    t = {}
    t['a'] = i
    l.append(t)
print(l)
t['a'] = 20
print(l)
print(map(id, l))

[{'a': 0}, {'a': 1}, {'a': 2}]
[{'a': 0}, {'a': 1}, {'a': 20}]
# they are all different!
[4533475600, 4533502592, 4533502872]

So no magic here

back to your question

This is what could be of interest for you: “Is python’s queue.Queue.put() thread safe?” meaning that the global variable q could be accessed by multiple concurrent threads safely. The answer is yes - it is thread safe

The Queue module implements multi-producer, multi-consumer queues. It is especially useful in threaded programming when information must be exchanged safely between multiple threads. The Queue class in this module implements all the required locking semantics

Oleg Kuralenko
  • 11,003
  • 1
  • 30
  • 40
2

In the first example, you are putting the same dict into the queue three times. This has nothing to do with the queue. You would find the same behaviour with list.append.

NSteinhoff
  • 301
  • 1
  • 3
  • But `put()` is called inside the for loop immediately following the modification of the dictionary. Shouldn't the modified dictionary be placed in the queue? – wrkyle Feb 11 '18 at 01:12
  • Ohhhh, I see what you mean. Three references to the same object are placed in the queue and that object has the final state after the loop, i.e., 'a'=2. – wrkyle Feb 11 '18 at 01:13