I was reading this question (which you do not have to read because I will copy what is there... I just wanted to give show you my inspiration)...
So, if I have a class that counts how many instances were created:
class Foo(object):
instance_count = 0
def __init__(self):
Foo.instance_count += 1
My question is, if I create Foo objects in multiple threads, is instance_count going to be correct? Are class variables safe to modify from multiple threads?
It's not threadsafe even on CPython. Try this to see for yourself:
import threading
class Foo(object):
instance_count = 0
def inc_by(n):
for i in xrange(n):
Foo.instance_count += 1
threads = [threading.Thread(target=inc_by, args=(100000,)) for thread_nr in xrange(100)]
for thread in threads: thread.start()
for thread in threads: thread.join()
print(Foo.instance_count) # Expected 10M for threadsafe ops, I get around 5M
The reason is that while INPLACE_ADD is atomic under GIL, the attribute is still loaded and store (see dis.dis(Foo.__init__)). Use a lock to serialize the access to the class variable:
Foo.lock = threading.Lock()
def interlocked_inc(n):
for i in xrange(n):
with Foo.lock:
Foo.instance_count += 1
threads = [threading.Thread(target=interlocked_inc, args=(100000,)) for thread_nr in xrange(100)]
for thread in threads: thread.start()
for thread in threads: thread.join()
print(Foo.instance_count)
No it is not thread safe. I've faced a similar problem a few days ago, and I chose to implement the lock thanks to a decorator. The benefit is that it makes the code readable:
def threadsafe_function(fn):
"""decorator making sure that the decorated function is thread safe"""
lock = threading.Lock()
def new(*args, **kwargs):
lock.acquire()
try:
r = fn(*args, **kwargs)
except Exception as e:
raise e
finally:
lock.release()
return r
return new
class X:
var = 0
@threadsafe_function
def inc_var(self):
X.var += 1
return X.var
Following on from luc's answer, here's a simplified decorator using with context manager and a little __main__ code to spin up the test. Try it with and without the @synchronized decorator to see the difference.
import concurrent.futures
import functools
import logging
import threading
def synchronized(function):
lock = threading.Lock()
@functools.wraps(function)
def wrapper(self, *args, **kwargs):
with lock:
return function(self, *args, **kwargs)
return wrapper
class Foo:
counter = 0
@synchronized
def increase(self):
Foo.counter += 1
if __name__ == "__main__":
foo = Foo()
print(f"Start value is {foo.counter}")
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
for index in range(200000):
executor.submit(foo.increase)
print(f"End value is {foo.counter}")
Without @synchronized
End value is 198124
End value is 196827
End value is 197968
With @synchronized
End value is 200000
End value is 200000
End value is 200000
Is modifying a class variable in python threadsafe?
While the Python GIL (Global Interpreter Lock) only allows access to one thread at a time, per atomic operation, some operations are not atomic, that is, they are implemented with more than one operation, such as, given (L, L1, L2 are lists, D, D1, D2 are dicts, x, y are objects, i, j are ints)
i = i+1
L.append(L[-1])
L[i] = L[j]
D[x] = D[x] + 1
See What kinds of global value mutation are thread-safe?
You're example is included in the non-safe operations, as += is short hand for i = i + 1.
Other posters have shown how to make the operation thread-safe. An alternative thread-safe way to implement your operation, without using a thread locking mechanism would be to reference a different variable, only set via an atomic operation. For example
max_reached = False
# in one thread
count = 0
maximum = 100
count += 1
if count >= maximum:
max_reached = True
# in another thread
while not max_reached:
time.sleep(1)
# do something
This would be thread safe, as long as only one thread increments the count.
I would say it is thread-safe, at least on CPython implementation. The GIL will make all your "threads" to run sequentially so they will not be able to mess with your reference count.
