should I protect built-in data structure( list, dict) when using multiple threads?

Question

I think I should use Lock object to protect custom class when using multiple threads, however, because Python use GIL to ensure that only one thread is running at any given time, does it mean that there's no need to use Lock to protect built-in type like list? example,

num_list = []
def consumer():
    while True:
        if len(num_list) > 0: 
            num = num_list.pop()
            print num
            return

def producer():
    num_list.append(1)

consumer_thread = threading.Thread(target = consumer)
producer_thread = threading.Thread(target = producer)
consumer_thread.start()
producer_thread.start()

Why dont you produce a working example..... that runs more than once.... — Nix, Feb 01 '13 at 16:09

score 1 · Answer 1 · edited May 23 '17 at 11:56

The GIL protects the interpreter state, not yours. There are some operations that are effectively atomic - they require a single bytecode and thus effectively do not require locking. (see is python variable assignment atomic? for an answer from a very reputable Python contributor).

There isn't really any good documentation on this though so I wouldn't rely on that in general unless if you plan on disassembling bytecode to test your assumptions. If you plan on modifying state from multiple contexts (or modifying and accessing complex state) then you should plan on using some sort of locking/synchronization mechanism.

If you're interested in approaching this class of problem from a different angle you should look into the Queue module. A common pattern in Python code is to use a synchronized queue to communicate among thread contexts rather than working with shared state.

Nix · Answer 2 · 2013-02-01T16:44:39.870

@jeremy-brown explains with words(see below)... but if you want a counter example:

The lock isn't protecting your state. The following example doesn't use locks, and as a result if the xrange value is high enough it will result in failures: IndexError: pop from empty list.

import threading
import time
con1_list =[]
con2_list =[]
stop = 10000
total = 500000
num_list = []
def consumer(name, doneFlag):
    while True:
        if len(num_list) > 0: 
            if name == 'nix':
                con2_list.append(num_list.pop())
                if len(con2_list) == stop:
                    print 'done b'
                    return
            else:
                con1_list.append(num_list.pop())
                if len(con1_list) == stop:
                    print 'done a'
                    return
def producer():
    for x in xrange(total):
        num_list.append(x)

def test():
    while not (len(con2_list) >=stop and len(con1_list) >=stop):
        time.sleep(1)
    print set(con1_list).intersection( set(con2_list))

consumer_thread = threading.Thread(target = consumer, args=('nick',done1))
consumer_thread2 = threading.Thread(target = consumer, args=('nix',done2))
producer_thread = threading.Thread(target = producer)
watcher = threading.Thread(target = test)
consumer_thread.start();consumer_thread2.start();producer_thread.start();watcher.start()

should I protect built-in data structure( list, dict) when using multiple threads?

2 Answers2