Removing items from list while iterating over it

Question

There are two separate processes running in Python script. Both interact with a global variable POST_QUEUE = []

Process 1 (P1) adds items to POST_QUEUE every 60 seconds. This can be anywhere from 0 to 50 items at a time.
Process 2 (P2) iterates over POST_QUEUE via a for-loop at set intervals and performs an operation on the list items one at a time. After performing said operation, the process removes the item from the list.

Below is a generalized version of P2:

def Process_2():
    for post in POST_QUEUE:
        if perform_operation(post):
            Print("Success!")
        else:
            Print("Failure.")
        POST_QUEUE.remove(post)

Understandably, I've run into an issue where when removing items from a list that a for-loop is iterating over, it screws up the indexing and terminates the loop earlier than expected (i.e., before it performs the necessary operation on each post and removes it from POST_QUEUE).

Is there a better way to do this than just creating a copy of POST_QUEUE and having P2 iterate over that while removing items from the original POST_QUEUE object? For example:

def Process_2():
    POST_QUEUE_COPY = POST_QUEUE[:]
    for post in POST_QUEUE_COPY:
        if perform_operation(post):
            Print("Success!")
        else:
            Print("Failure.")
        POST_QUEUE.remove(post)

Does this answer your question? [Are Python built-in containers thread-safe?](https://stackoverflow.com/questions/2227169/are-python-built-in-containers-thread-safe) — JulienD, Dec 27 '20 at 19:16
Is the issue resolved? If so, please mark the correct answer as accepted or post and accept a solution of your own. — Jarvis, Dec 28 '20 at 01:41

IoaTzimas · Answer 1 · 2020-12-27T19:35:03.290

You can loop through your list from right to left. This way, the removal of items will not cause issues to the loop. It's not a good idea in general to remove items from list while looping, but if you need to do it, going from right to left is the best option:

def Process_2():
    for i in range(len(POST_QUEUE)-1, -1, -1):
        if perform_operation(POST_QUEUE[i]):
            Print("Success!")
        else:
            Print("Failure.")
        POST_QUEUE.pop(i)

Jarvis · Answer 2 · 2020-12-27T19:19:13.407

0

How about this:

while POST_QUEUE_COPY:
    post = POST_QUEUE_COPY.pop(0)
    if perform_operation(post):
        Print("Success!")
    else:
        Print("Failure.")

And processes don’t share data, threads do. So unless you are using something like multiprocessing.Manager or some shared-memory construct, I don’t think your current logic would work.

edited Dec 27 '20 at 19:19

answered Dec 27 '20 at 19:13

Jarvis

8,494
3
27
58

score 0 · Accepted Answer · answered Dec 27 '20 at 19:38

Since you do not really need the indexes of the elements I would suggest something like this as an easy solution:

def Process_2():
    while len(POST_QUEUE):
        if perform_operation(post[0]):
            Print("Success!")
        else:
            Print("Failure.")
        POST_QUEUE.remove(post[0])

However this solution has a runtime of O(n^2) for every use of the loop since python needs to move every element in the list on every iteration.

So IMO a better implmentation would be:

def Process_2():
    reversed_post_queue = POST_QUEUE[::-1]
    while len(reversed_post_queue):
        if perform_operation(post[-1]):
            Print("Success!")
        else:
            Print("Failure.")
        POST_QUEUE.remove(post[-1])

that way you keep the order (which I suppose is important to you throughout this answer) while only moving the elements of the list once and resulting in a runtime of O(n)

finally, the best implementation IMO is to create or import a queue module so that you could easily use the list as FIFO.

Removing items from list while iterating over it

3 Answers3