Is there a way to perform test-and-set operation on dict in python?

Question

I have a multithreaded app in which I need to use a global dict to remember and handle requests.

The problem is, I am not supposed to perform the same request twice, so I need dict to be able to test if the key exists. if not, insert it, if it does, report error.

All code runs in python so GIL should not be relevant.

Is that feasible in python?

Here lock would work but I am trying to find a more natural way to solve it.

Dr. Jan-Philip Gehrcke · Accepted Answer · 2015-02-09T15:08:46.147

0

You can surely create a dictionary and share it among the threads, but you should protect each access to the dictionary (be it read or write) with a lock.

Here lock would work but I am trying to find a more natural way to solve it.

The usage of synchronization primitives such as locks is the natural way to implement a reliable architecture involving concurrency.

That being said, making sure that a certain request is not made twice can be approached on various levels. If you are doing it with a dictionary like described, you need to insert the identifier right before performing the request. Pseudo code:

with lock.acquire():
    if url in d:
        return # do not perform request
    d.[url] = True

perform_request(url)

If the request failed, you might want to remove the key from dict again (not without acquiring the lock, of course).

edited Feb 09 '15 at 15:08

answered Feb 09 '15 at 15:03

Dr. Jan-Philip Gehrcke

33,287
14
85
130

so you mean there is no such atomic operation in dict – Jason Hu Feb 09 '15 at 15:32
There is no *guarantee* for dict operations to be atomic/thread-safe, so do not rely on it. Also see http://stackoverflow.com/questions/3358770/python-dictionary-is-thread-safe – Dr. Jan-Philip Gehrcke Feb 09 '15 at 15:59
There might be Python implementations (certain CPython versions, or PyPy or IronPython) where certain dict operations are thread-safe, but as long as this is not documented, it is an implementation detail that you should not build your architecture on. – Dr. Jan-Philip Gehrcke Feb 09 '15 at 16:08
fair enough answer. it seems it's not a good idea to abuse the language. – Jason Hu Feb 09 '15 at 21:35
Right. And as I said -- there really is nothing "unnatural" about explicitly taking care of synchronization. :-). – Dr. Jan-Philip Gehrcke Feb 10 '15 at 18:23

Is there a way to perform test-and-set operation on dict in python?

1 Answers1