4

i am programming a telegram bot where more thread handles every update async. i hard code an empty dictionary and while the bot is running, multiple threads write and read from it.

The most common operation is:

if key not in dict:
    dict[key] = value
else:
    dict[key].append(another_value)

the problem is that while a thread check if key is or not in dict, and maybe it is not, another thread wrote the key in it.

so basically i have to get rid of this racing condition.

i need a fast solution. searching on the web i found answers talking about threading.Lock(). Those answers are about 10 years old. i wonder if nowadays it's still a good solution or maybe there is some more new library

91DarioDev
  • 1,612
  • 1
  • 14
  • 31
  • Related: https://stackoverflow.com/questions/17682484/is-collections-defaultdict-thread-safe – Robᵩ Jan 03 '18 at 00:26

2 Answers2

5

You can use a defaultdict or, if you want to use a regular dict, the setdefault() method.

Using defaultdict:

from collections import defaultdict
mydict = defaultdict(list)    # new key is initialized with an empty list
mydict[key].append(value)     # so it is always safe to append

Using setdefault():

mydict = {}
mydict.setdefault(key, []).append(value)

Either should be thread-safe, IIRC, even if they don't look that way: the method implementations are in C and therefore can't be interrupted by Python code. So it is never possible to, for example, have two setdefault() methods executing at literally the same time and causing a race condition.

Generally defaultdict is preferred because it is more performant, makes empty lists only when needed, and the resulting code is simpler. The setdefault() version could be better if some keys in your dictionary might have non-list values.

kindall
  • 178,883
  • 35
  • 278
  • 309
  • sorry but even if they are not executing at the same time, since i have to first check if the key exists, and if not i add it, isn't it possible that after the check (key is not present) and before i add the key, another thread already added the key so the first thread is trying to adding it again? – 91DarioDev Jan 03 '18 at 00:39
  • No, you don't have to check if the key exists. That's the whole point! – kindall Jan 03 '18 at 00:41
  • ah nice. so i directly append data (list) to a key, not caring if it exists or not, but it will be automatically added if it doesn't exist? but can i also know if the element i am appending is the first one or not in a thread safe way? – 91DarioDev Jan 03 '18 at 00:46
  • Yes, that's exactly what happens. Unfortunately there's not any way to tell whether an item is the first or not without using a lock. – kindall Jan 03 '18 at 00:49
  • ah ok. so if i need it, i should go for the threading.Lock() solution? thank you very much – 91DarioDev Jan 03 '18 at 00:51
  • Yes, I'd use a proper `Lock`. – kindall Jan 03 '18 at 00:57
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/162383/discussion-between-91dariodev-and-kindall). – 91DarioDev Jan 03 '18 at 00:57
0

Python's dicts are safe for single operations, but multiple operations are not. What you're currently doing is multiple operations. Locks are fine, and generally the standard way to handle this problem.

Take a look here to see some of the details regarding Python's thread-safety.

http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm

ollien
  • 4,418
  • 9
  • 35
  • 58