Unexpected IndexError while removing list items

Question

I am a beginner in Python. I have learned other languages before, such as C++ (beginner) and JQuery. But I find the looping in python is quite confusing.

Well I want to achieve a simple result. The program will loop through a list of words, and then it will remove the words that match with the first two letters with the next word in the list:

test = ['aac', 'aad', 'aac', 'asd', 'msc']
for i in range(len(test)):
    if test[i][0:2] == test[i+1][0:2]:
        test.remove(test[i])

# This should output only ['aac', 'asd', 'msc']
print test

The code above should remove 'aac' and 'aad' from the list. But in reality, this raises an IndexError. Furthermore, I wasn't able to achieve the desired result. Can you please explain?

Answer 1

You are changing the length of the list while looping over a range that goes up to the starting length of the list; remove one item from the list and the last index is no longer valid.

Moveover, because items are removed from the list at the current index, the rest of the list indices shift; what was at index i + 1 is now at index i and your loop index is no longer useful.

Last but not least, you are looping until the very last index of test, but then try to access test[i + 1] still; that index does not exist even if you were not removing elements from the list.

You could use a while loop to achieve what you want to do:

test = ['aac', 'aad', 'aac', 'asd', 'msc']
i = 0
while i < len(test) - 1:
    if test[i][:2] == test[i+1][:2]:
        del test[i]
        continue
    i += 1

Now i is tested against the new length each loop iteration, and we only increment i if no element was removed. Note that the loop is limited to the length minus 1 because you want to test for test[i + 1] each iteration.

Note that I use del test[i]; no need to scan through the list searching for that the value-to-remove again; this could lead to subtle bugs as well if values appear multiple times in the list but only later instances should be removed; e.g. ['aac', 'foo', 'aac', 'aad'] should result in ['aac', 'foo', 'aad'], not ['foo', 'aac', 'aad'], which is what test.remove(test[i]) would result in.

Demo:

>>> test = ['aac', 'aad', 'aac', 'asd', 'msc']
>>> i = 0
>>> while i < len(test) - 1:
...     if test[i][:2] == test[i+1][:2]:
...         del test[i]
...         continue
...     i += 1
... 
>>> test
['aac', 'asd', 'msc']

You could use a list comprehension to avoid the shrinking list problem:

>>> [t for i, t in enumerate(test) if i == len(test) - 1 or t[:2] != test[i + 1][:2]]
['aac', 'asd', 'msc']

Both approaches require only one loop through the input list.

Answer 2

As you removing items from the list, range(len(test)) still holds the same value. So even if your test list has only no items left, the loop is still going.

I have two solutions:

1. Copy the items you want to a new list, so instead of deleting it:

   test2 = test[i]
   

   And don't forget to reverse the conditions.

2. Loop it backwards. Like this:

   n = len(test)
   for i in range(n):
       j = n - i - 1
       if j > 1:
       if test[j][0:2] == test[j-1][0:2]:
           test.remove(test[j])
   

   Or, as martijn suggested:

   n = len(test)
   for i in range(n-1, 0, -1):
       if i > 1:
       if test[i][0:2] == test[i-1][0:2]:
           test.remove(test[i])
   

Hope it helps!

P.S sorry for my stupid, previous answer

Answer 3

As others have said as you remove items the list gets shorter causing an index error.

Keeping in line with the original question. if your looking to remove items using list.remove() you can add the found items to a list then iterate over them and remove them from your original list like so:

# Set up the variables
test = ['aac', 'aad', 'aac', 'asd', 'msc']
found = []
# Loop Over the range of the lenght of the set
for i in range(len(test)):
    try:
        if test[i].startswith(test[i+1][0:2]):
            found.append(test[i])  # Add the found item to the found list
    except IndexError: # You'll hit this when you do test[i+1]
        pass

# Remove the Items at this point so you don't cause any issues
for item in found:
    test.remove(item)  # If an item has been found remove the first instance

# This sholuld output only ['aac', 'asd', 'msc']
print test

EDIT:

As per Martins comment, you don't need to make a second list of items that need to be removed you can instead make a list of items that didn't need to be removed like so:

# Set up the variables
test = ['aac', 'aad', 'aac', 'asd', 'msc']
found = []

# Loop Over the range of the lenght of the set
for i in range(len(test)):
    try:
        if not test[i].startswith(test[i+1][0:2]):
            found.append(test[i])  # Add the found item to the found list
    except IndexError: # You'll hit this when you do test[i+1]
        found.append(test[i]) # If there is no test[i+1], test[i] must be cool.


# This sholuld output only ['aac', 'asd', 'msc']
print found

Answer 4

for i in range(len(test)) gives you a list with the valid indices of test. However, as you keep deleting items from test in the loop, the size of test reduces, causing some of those originally valid indices to become invalid.

What you're doing is something like this:

L = range(len(test))
for i in L:
  if condition:
    # remove something from test <- the size of test has changed.
                                 # L[-1] is no longer a valid index in test

What you could do instead, is to accumulate the indices of things that you would like to delete and delete them later:

deleteThese = set()
for i,item in enumerate(test[:-1]):
  if item[0:2] == test[i+1][0:2]:
    deleteThese.add(i)
test = [item for i,item in enumerate(test) if i not in deleteThese]

Output:

In [70]: test = ['aac', 'aad', 'aac', 'asd', 'msc']

In [71]: %paste
deleteThese = set()
for i,item in enumerate(test[:-1]):
  if item[0:2] == test[i+1][0:2]:
    deleteThese.add(i)
test = [item for i,item in enumerate(test) if i not in deleteThese]

## -- End pasted text --

In [72]: test
Out[72]: ['aac', 'asd', 'msc']