Python: Will A List Changing Size During 'not in' Cause UB?

Question

I have a hypothetical program with two threads.

items = [ 'A', 'B' ]


func1():
    while True:
        if 'C' not in items:
            # do something
        
func2():
    while True:
        items.clear()
        items.append('A')
        items.append('B')

main():
    start_new_thread(func1)
    start_new_thread(func2)

When not in is executed, I assume Python must internally iterate over the list to check each element. If func2 changes the size of this list during the iteration, will it cause an exception or UB?

Welcome to StackOverflow. It is very difficult to debug code without being able to read any of it. Check out this article for writing good questions on stack overflow. https://stackoverflow.com/help/how-to-ask — Andrew-Harelson, Oct 02 '21 at 01:14
@Andrew-Harelson There is no bug. This is a question about the Python programming language, not a bug in a specific program. — Abraham, Oct 02 '21 at 01:17
A lot does depend on the specifics; is thread 2 changing the list, or replacing it? Is it literally a list, or another list-like data structure? — Jiří Baum, Oct 02 '21 at 01:27
Fair enough, that being said, showing is always better than telling. You should try to write a simple program to demonstrate the question you are trying to ask. Explaining how a hypothetical program works will lead to confusion. This is why you are getting lots of downvotes on this question. — Andrew-Harelson, Oct 02 '21 at 01:28
@Andrew-Harelson I edited the question with a minimal example, hope that's better — Abraham, Oct 02 '21 at 01:36

score 0 · Accepted Answer · answered Oct 02 '21 at 01:56

0

You should never allow two threads to access the same piece of memory, at the same time, in any programming language, ever. This creates a data race condition which leads to extremely undefined behavior (If your programming language of choice cannot detect this and throw an exception). The safest thing to do is use a lock object to force the execution of a thread to stop if another thread has acquired the lock, and wait until it is safe to modify the data again. Here is your minimal example with locks added (and also modified to be actual working code).

import threading

items = [ 'A', 'B' ]
lock = threading.Lock()

def func1():
    while True:
        lock.acquire()
        if 'C' not in items:
            print("C not in items")
        lock.release()

def func2():
    while True:
        lock.acquire()
        items.clear()
        items.append('A')
        items.append('B')
        lock.release()

def main():
    thread_a = threading.Thread(target=func1, daemon=True)
    thread_b = threading.Thread(target=func2, daemon=True)
    thread_a.start()
    thread_b.start()
    thread_a.join()
    thread_b.join()

main()

Note that many libraries may refer to some of their functions as being "thread safe". This means that they already handled any potential data race conditions and you do not have to worry about setting up locks. You may also encounter "atomic" operations, which basically just means that they are thread-safe operations. Atomic operations aren't really a thing in python though.

answered Oct 02 '21 at 01:56

Andrew-Harelson

1,022
3
11

What about the GIL? – no comment Oct 02 '21 at 02:01
1

The GIL just guarantees that only one thread runs at a time, it doesn't guarantee that threads won't swap in the middle of checking a list with the `not in` operator. Granted the interpreter lock prevents a lot of these sorts of issues, it cannot prevent all of them. – Andrew-Harelson Oct 02 '21 at 02:13
But it's just list and strings, that's all C code, no? – no comment Oct 02 '21 at 02:15
Yes python implements lists and strings with C. How does that affect thread safety? – Andrew-Harelson Oct 02 '21 at 02:18
The GIL prevents thread switching during C code, no? – no comment Oct 02 '21 at 02:21
Not necessarily, for some types the C code will need to call `__contains__` and hook back into python, which could unlock the interpreter. You're right that for built-in types this will never happen, but it is unsafe to assume you will always have a list with thread safe types. These posts might explain itthis better than me. https://stackoverflow.com/questions/19727759/is-pythons-in-language-construct-thread-safe-for-lists https://stackoverflow.com/questions/6319207/are-lists-thread-safe – Andrew-Harelson Oct 02 '21 at 02:57

Python: Will A List Changing Size During 'not in' Cause UB?

1 Answers1