Python Remove Duplicates From list of tuples?

Question

Please Note, question got fixed.

In python I have the following list

[('admin', 'admin', None), ('admin', 'admin', None), ('admin', 'admin', 1)]

How can I remove duplicates where I consider an item to be duplicate if it has the same 3 first tuples, so I'm expecting to get the following output:

[('admin', 'admin', None), ('admin', 'admin', 1)]

as we can find ('admin', 'admin', None) twice in the original input.

The provided and only answer fails on the following input:

[('admin', '', {'type': 'http_basic', 'login_url': 'https://IP_HERE/AuthChk', 'method': 'get'})]

While I'm expecting it to return exactly same list

Same error for:

list(set([i for i in lst]))

Does this answer your question? https://stackoverflow.com/questions/480214/how-do-you-remove-duplicates-from-a-list-whilst-preserving-order — Léo Beaucourt, Apr 13 '22 at 08:44
If duplicate has same first 2 elements, why does your expected output look like it does? — matszwecja, Apr 13 '22 at 08:44
Your result is wrong - all your results have the same 2 first elemets — Patrick Artner, Apr 13 '22 at 08:44
`list( set( [('admin', 'admin', None), ('admin', 'admin', None), ('admin', 'admin', 1)] )` ) would give you what you want ... (not order-wise but element-wise) it takes all elements into account. — Patrick Artner, Apr 13 '22 at 08:46
@PatrickArtner sorry for the mistake I did, question got updated. please keep your comment for the previous version as it's helpful as well — John, Apr 13 '22 at 08:48
Expected output makes even less sense now. you removed the non-duplicate — matszwecja, Apr 13 '22 at 08:50
I'm completely lost at this point. None of those has 3 same elements and looks like you reverted your initial correction instead of fixing something else. — matszwecja, Apr 13 '22 at 08:58
Yeah, all of them has the same 2 values - 'admin' and 'admin'. And you ask about "same 3 first fields". — matszwecja, Apr 13 '22 at 09:00
I mean look at the first 2 tuples... they are the same so they are replaced by one copy... — John, Apr 13 '22 at 09:11

score 0 · Answer 1 · answered Apr 13 '22 at 09:38

0

Try this:

result = []
first_timers = set()
for i in lst:
    if hash(str(i)) not in first_timers:
        first_timers.add(hash(str(i)))
        result.append(i)

answered Apr 13 '22 at 09:38

Nikolay Tsvetanov

94
7

Patrick Artner · Answer 2 · 2022-04-13T09:43:31.257

Your problem - which was not adequately captured by your simplified example of

[('admin', 'admin', None), ('admin', 'admin', None), ('admin', 'admin', 1)]

is that your tuple contains non-hash-able datatypes (aka: dict's / set's etc.).

You can transform your inner tuples to stringified versions of it, put them into a dict under that key with the original tuple as value - then use the dict.values() as outputs:

data = [('admin', '', {'type': 'http_basic', 'login_url': 'https://IP_HERE/AuthChk', 'method': 'get'}),
    ('admin', '', {'type': 'http_basic', 'login_url': 'https://IP_HERE/AuthChk', 'method': 'get'}),
    ('nonadmin', '', {'type': 'http_basic', 'login_url': 'https://IP_HERE/AuthChk', 'method': 'get'})]

def strKey(whatever):
    return tuple(map(str, whatever))

d = { strKey(inner):inner for inner in data}

uniqified = list(d.values())
print(uniqified)

Output:

[('admin', '', {'type': 'http_basic', 'login_url': 'https://IP_HERE/AuthChk', 'method': 'get'}),
 ('nonadmin', '', {'type': 'http_basic', 'login_url': 'https://IP_HERE/AuthChk', 'method': 'get'})]

Python Remove Duplicates From list of tuples?

2 Answers2