0

I am attempting to remove duplicates from a list of numbers; however, the code doesn't remove all the duplicates. After debugging, I realized that an integer is skipped if the previous iteration results in the removal of that integer (neither of the 2's in the list are removed as they are preceded by 5 which was a duplicate):

numbers_list = [5, 2, 546, 7, 3, 5, 2, 7, 29, 6, 5, 7]
for item in numbers_list:
    if numbers_list.count(item) > 1:
        numbers_list.remove(item)
    else:
        pass
print(numbers_list)

EDIT: I know there are other ways to remove duplicates from a list but I want to know how to ensure that iterations aren't skipped.

ami
  • 1
  • 3
  • 1
    Check this answer: https://stackoverflow.com/a/1207461/6317556 – Alexandru Jul 23 '20 at 10:30
  • @Alexandru thank you, but this doesn't answer my question - I have edited my question to make it more clear – ami Jul 23 '20 at 10:39
  • You should NOT remove items from the list that you are iterating over, the iterations are skipped because you are breaking the "for" iteration process by changing the list while looping on it. – João Pinto Jul 23 '20 at 14:58

4 Answers4

0

If you want to remove duplicates you can do:

mylist = list(dict.fromkeys(mylist))

If you don't want to skip iterations you can use a while, for example:

counter = 0
while counter < len(mylist):
  # if you want to remove an item, don't increase counter, otherwise yes
  counter++
Alexandru
  • 143
  • 9
0

if you need to remove duplicates you can use this

numbers_list = [5, 2, 546, 7, 3, 5, 2, 7, 29, 6, 5, 7]
list_no_dubl = list(set(numbers_list))
print(list_no_dubl)
DD_N0p
  • 229
  • 1
  • 2
  • 6
0

Just don't modify a list during iteration -- strange things happen, as you observed. Changing the list affects the iterator, and the results of that are, I think, implementation-defined. Doing it this way is considered bad style, exactly because of those weird effects. The easiest solution for your concrete problem of removing duplicates, which is equivalent to ensuring uniqueness, is to use a set:

unique_list = list(set(numbers_list))

You can leave out the list part if you are only interested in the result as an iterable. However, this (likely) won't preserve the orginal order in numbers_list -- if you need the order, some different approach is needed, like this:

def makeunique(l): 
   i = 0 
   while i < len(l): 
       j = i + 1  
       while j < len(l): 
           if l[j] == l[i]: 
               del l[j] 
           j += 1 
       i += 1 
   return l

(This has quadratic complexity, though.)

phipsgabler
  • 20,535
  • 4
  • 40
  • 60
  • thank you. So there is no way to prevent the skipping? Yes, I need the order but I believe using `dict.fromkeys` will preserve the order. – ami Jul 23 '20 at 10:44
  • Not really, AFAIK, unless you know exactly how changing the list affects the iterator, which is, I think, implementation-defined. Doing it this way is considered bad style, exactly because of those weird effects. The real solution is constructing a new list (e.g. with `set`), or indexing instead of an iterator. – phipsgabler Jul 23 '20 at 10:49
0

The function set() constructs a set object, which ensures that there are only unique, unordered collection of objects.

The function list() converts the set object back into a list.

numbers_list = [5, 2, 546, 7, 3, 5, 2, 7, 29, 6, 5, 7]
deduped_list = list(set(numbers_list))
print(deduped_list)
>>[2, 3, 546, 5, 6, 7, 29]
Ian
  • 933
  • 12
  • 17