Python: remove duplicate items from a list while iterating

Question

I have a list named results, I want to get rid of the duplicate items in the list, the expected result and my results does not match, I checked my code but cannot find what is the problem, what happened? Thanks a lot!

results = [[-4, -2, 6], [-4, -2, 6], [-4, -2, 6], [-4, -2, 6], [-4, -2, 
    6], [-4, -2, 6], [-4, 0, 4], [-4, 0, 4], [-4, 1, 3], [-4, 1, 3], [-4, 2, 
    2], [-4, 2, 2], [-4, 2, 2], [-2, -2, 4], [-2, -2, 4], [-2, -2, 4], [-2, 
    -2, 4], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, -2, 4], [-2, -2, 4], 
    [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2]]

i = 0
while i < len(results):  
    j = i+1
    while j < len(results):
        if(set(results[i]) == set(results[j])):
            results.remove(results[j])
        else:
            j = j+1
    i = i+1
print(results)

OUTPUT:
[[-4,-2,6],[-4,0,4],[-4,1,3],[-4,2,2],[-2,-2,4],[-2,-2,4],[-2,0,2]]

EXPECTED RESULT:
[[-4,-2,6],[-4,0,4],[-4,1,3],[-4,2,2],[-2,-2,4],[-2,0,2]]

UPDATE: I got it. no problem with the logic of this code but I made a simple mistake with one place (sorry...I am a newbie). I should replace method "remove" by method "del", because I want to remove the item with specified index, if use "remove", it always remove the first one shows up in the list of that value. Anyway, Thanks to all!

For example:
myList = ['Apple', 'Banana', 'Carrot','Apple']
myList.remove(myList[3])
print(myList)
expected output:['Apple', 'Banana', 'Carrot']
actual output: ['Banana', 'Carrot', 'Apple']

myList = ['Apple', 'Banana', 'Carrot','Apple']
del (myList[3])
print(myList)
OUTPUT: ['Apple', 'Banana', 'Carrot']

SOLUTION to my question:

### use "del" instead of "remove"
#results.remove(results[j])
del results[j]

Another simple test example similar to my original question:
results = [[-2, -2, 4], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, -2, 4]]
i = 0
while i < len(results):  
    j = i+1
    while j < len(results):
        print(results[i],results[j])
            if(set(results[i]) == set(results[j])):
                #would get incorrect answer with "replace"
                results.remove(results[j])
                #try "del" to get the correct answer
                #del (results[j]) 
    else:
        j = j+1
i = i+1
print(results)

You're iterating over an object while mutating it, which can be tricky. Try iterating over it backwards. — Jared Goguen, Apr 20 '17 at 20:34
Do not remove from existing list, it will be better to create new list and add unique items to it. — Uladzimir Palekh, Apr 20 '17 at 20:34
If you really want to understand what is wrong with your code, it seems to me that setting j to i+1 is the mistake here. You want j to take all values but i, so j must go from 0 to your length and skip only current i value. Now as the other answers state, you have more pythonic ways to achieve duplicates removal. — Tanguy A., Apr 20 '17 at 20:44

score 3 · Answer 1 · edited May 23 '17 at 12:17

3

Thanks to Remove duplicate sublists from a list

results = [[-4, -2, 6], [-4, -2, 6], [-4, -2, 6], [-4, -2, 6], [-4, -2, 
    6], [-4, -2, 6], [-4, 0, 4], [-4, 0, 4], [-4, 1, 3], [-4, 1, 3], [-4, 2, 
    2], [-4, 2, 2], [-4, 2, 2], [-2, -2, 4], [-2, -2, 4], [-2, -2, 4], [-2, 
    -2, 4], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, -2, 4], [-2, -2, 4], 
    [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2], [-2, 0, 2]]

results = [list(x) for x in set([tuple(x) for x in results])]
print (results)

prints:

[[-4, -2, 6], [-2, 0, 2], [-4, 1, 3], [-2, -2, 4], [-4, 2, 2], [-4, 0, 4]]

edited May 23 '17 at 12:17

Community

1
1

answered Apr 20 '17 at 20:36

ragardner

1,836
5
22
45

thank you ,this may be an easier way to do it. BTW, I find the problem with my own code. – Xuyong Apr 21 '17 at 16:49

score 3 · Answer 2 · answered Apr 20 '17 at 20:38

3

The following loop will iterate backwards over the pairs, allowing you to mutate the list during iteration.

i = len(results) - 1
while i >= 1:  
    j = i - 1
    while j >= 0:
        if(set(results[i]) == set(results[j])):
            results.remove(results[j])
            i -= 1
        j -= 1
    i -= 1

But, yes, using sets would probably be better. A functional version of the above would be:

map(list, set(map(tuple, results)))

answered Apr 20 '17 at 20:38

Jared Goguen

8,772
2
18
36

it is interesting to do backwards to avoid the problem, your answer may be most near to what I want. thank you ,BTW, I find the problem with my own code. – Xuyong Apr 21 '17 at 16:50

score 2 · Answer 3 · edited May 23 '17 at 12:34

Set-ify everything.

Convert the inner lists to sets to remove duplicate elements within them. Push all these sets into another set to get rid of duplicate (sub)sets. Finally, convert everything back to a list of lists:

no_dups = [list(s) for s in set(frozenset(l) for l in results))]

I refered to How can I create a Set of Sets in Python? to figure out the bit about frozenset. You can't just convert the sublists to sets since the outer set needs immutable elements to hash (and regular sets are mutable). Of course, you could get around this by using maps/tuples as other answers point out.

score 1 · Answer 4 · answered Apr 20 '17 at 20:46

1

You may use set with map as:

my_uniques = set(map(tuple, results))
#                     ^ type-cast inner list to tuples
#                       because lists are not hashable and
#                       hence can't be used with set

where my_uniques will be a set of unique tuple objects:

set([(-2, -2, 4), (-4, 0, 4), (-4, -2, 6), (-4, 2, 2), (-2, 0, 2), (-4, 1, 3)])

If you want to convert it to list of lists (which I don't think is necessary), you have to explicitly do:

my_uniques  = list(map(list, my_uniques))

answered Apr 20 '17 at 20:46

Moinuddin Quadri

46,825
13
96
126

thank you , I tried several test cases, but sometimes it still has duplicate item, I did some change to my original code and it works. anyway, thanks very for your help, your method could be a easier way to achieve my goal. – Xuyong Apr 21 '17 at 16:53
I strongly doubt about the duplicate tuples in the set. But it is worth mentioning that tuple `(0,0,1)` and `(1,0,0)` are different i.e. order matters. In case for you this order doesn't matter, you should be using `sorted()` function on each tuple which will remove the tuples with same number but with different positions from the `set` object – Moinuddin Quadri Apr 21 '17 at 16:57

Python: remove duplicate items from a list while iterating

4 Answers4

Linked