How to remove unhashable duplicates from a list in Python?

Question

My data is this:

[{u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': [u'/www/web'], u'server_port': u'80'}, {u'webpath': [u'/www/web'], u'server_port': u'80'}, {u'webpath': [u'/www/shanghu'], u'server_port': u'80'}, {u'webpath': [u'/www/shanghu'], u'server_port': u'80'}, {u'webpath': [u'/www/www/html/falv'], u'server_port': u'80'}, {u'webpath': [u'/www/www/html/falv'], u'server_port': u'80'}, {u'webpath': [u'/www/www/html/falv'], u'server_port': u'80'}, {u'webpath': [u'/www/falvhezi'], u'server_port': u'80'}, {u'webpath': [u'/www/test10'], u'server_port': u'80'}, {u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': u'/etc/html', u'server_port': u'80'}, {u'webpath': [u'/www/400.ask.com'], u'server_port': u'80'}, {u'webpath': [u'/www/www'], u'server_port': u'80'}, {u'webpath': [u'/www/www'], u'server_port': u'80'}, {u'webpath': [u'/www/www'], u'server_port': u'80'}, {u'webpath': [u'/www/zhuanti'], u'server_port': u'80'}, {u'webpath': [u'/www/zhuanti'], u'server_port': u'80'}, {u'webpath': [u'/www/shanghu'], u'server_port': u'80'}]

My code is this:

    seen = set()
    new_webpath_list = []
    for webpath in nginxConfs:
        t = tuple(webpath.items())
        if t not in seen:
            seen.add(t)
            new_webpath_list.append(webpath)

But the script returns:

TypeError: "unhashable type: 'list'"

possible duplicate of [Remove duplicate dict in list in Python](http://stackoverflow.com/questions/9427163/remove-duplicate-dict-in-list-in-python) — muddyfish, Jul 23 '15 at 13:48
@muddyfish it doesn't apply. the same error shall be raised. `.items()` doesn't turn list into tuple. — Jason Hu, Jul 23 '15 at 13:50
You are creating tuples, but there can still be un-hashable lists inside those tuples! — tobias_k, Jul 23 '15 at 13:51

tobias_k · Accepted Answer · 2015-07-23T14:03:10.113

You are creating tuples from the dictionaries to make them hashable, but there can still be non-hashable lists inside those tuples! Instead, you also have to "tuplefy" the values.

t = tuple(((k, tuple(v)) for (k, v) in webpath.items()))

Note that this is a bit glitchy as the first entry in the dict is just a string, while the others are lists of strings. You could mend this with an if/else, but it should not really be necessary.

t = tuple(((k, tuple(v) if isinstance(v, list) else v) for (k, v) in webpath.items()))

Alternatively, you could also just memorize the string represenations of the dictionaries...

t = repr(webpath)

John Y · Answer 2 · 2015-07-23T15:25:11.073

The most straightforward way to do this is to just test membership directly using the new list you are building.

new_webpath_list = []
for webpath in nginxConfs:
    if webpath not in new_webpath_list:
        new_webpath_list.append(webpath)

This handles the cases where there is an arbitrary (unknown beforehand) level of nesting of unhashable types. It also makes your code simpler, easier to understand, and very possibly more efficient, because you are not creating extra data that you don't need (no seen set, no conversion of elements to tuples).

Pedro Lobito · Answer 3 · 2020-06-05T10:19:23.843

0

Late answer, but I was able to remove duplicated dict from a list using:

old_list = [{"x": 1}, {"x": 1}, {"x": 2}]
new_list = []
[new_list.append(x) for x in old_list if x not in new_list]
# [{'x': 1}, {'x': 2}]

Demo

edited Jun 05 '20 at 10:19

answered Jun 05 '20 at 01:44

Pedro Lobito

94,083
31
258
268

How to remove unhashable duplicates from a list in Python?

3 Answers3