what's the best practice for accessing shared data with python multi-threading

Question

In python multi-threading, there are some atomic types that can be accessed by multiple threads without protection(list, dict, etc). There are also some types need protected by lock.

My question is:

1. where can I find official document that list all atomic types, I can google some answers, but they are not "official" and out of date.

2. some book suggest that we should protect all shared data with lock, because atomic type may because non-atomic, we shouldn't rely on it. Is this correct? because lock surely have overhead, is this overhead negligible even with big program?

Answer 1

Locks are used for making an operation atomic. This means only one thread can access some resource. Using many locks causes your application lose the benefit of threading, as only one thread can access the resource.

If you think about it, it doesn't make much sense. It will make your program slower, because of the python needs to manage and context switch between the threads.

When using threads, you should look for minimizing the number of locks as much as possible. Try use local variables whenever possible. Make your function do some work, and return a value instead of updating an existing one. Then you can create a Queue and collect the results.

Besides locks, there are Semaphores. These are basically Locks, with a limited number of threads can use it:

A semaphore manages an internal counter which is decremented by each acquire() call and incremented by each release() call. The counter can never go below zero; when acquire() finds that it is zero, it blocks, waiting until some other thread calls release().

Python has a good documentation for threading module.

---

Here is a small example of a dummy function tested using single thread vs 3 threads. Pay attention to the impact Lock makes on the running time:

threads (no locks) duration: 1.0949997901

threads (with locks) duration: 3.1289999485

single thread duration: 3.09899997711

def work():
    x = 0
    for i in range(100):
        x += i
    lock.acquire()
    print 'acquried lock, do some calculations'
    time.sleep(1)
    print x
    lock.release()
    print 'lock released'

Answer 2

I think you are looking for this link.

From above link :

An operation acting on shared memory is atomic if it completes in a single step relative to other threads. When an atomic store is performed on a shared variable, no other thread can observe the modification half-complete. When an atomic load is performed on a shared variable, it reads the entire value as it appeared at a single moment in time. Non-atomic loads and stores do not make those guarantees.

Any manipulation on list won't be atomic operation, so extra care need to be taken to make it thread safe using Lock, Event, Condition or Semaphores etc.

For example, you can check this answer which explains how list are thread safe.