Remove duplicates from a nested dictionary

Question

I have a dictionary as follows:

   data =` {0: {key1: {key2 : [value1 , value2 ,value3]}}}`

Now I have to remove the duplicate value from the list, which is inside the nested dictionary. Also, the function should be recursive and should pass two arguments, like :

remove_duplicates(dict , 'key2')


def remove_duplicate(dict, keys):
    for key, value in duplicate_values_dict.items():
        if key in dict_keys:
            duplicate_values_dict[key] = set(value)
     return duplicate_values_dict

Here, I am unable to pass tuple of key, as in the question.

The function should remove the duplicate value from the specified key and append it in a new dictionary.

How do I solve the problem?

you want to remove duplicates from the list inside the nested dictionary ? — venkata krishnan, Jul 29 '19 at 04:19
Check this link https://stackoverflow.com/questions/7961363/removing-duplicates-in-lists/7961390 — Sukma Saputra, Jul 29 '19 at 04:28
the list in the 3rd level of dictionary has duplicate values, that needs to be removed, but the function should be recursive — Gurpreet Singh Kaptan, Jul 29 '19 at 04:47

score 0 · Accepted Answer · edited Aug 02 '19 at 01:07

0

The data you have shared contains nested dictionary, So I make a recursive function:

def remove_duplicate(duplicate_values_dict, dict_keys):
    if dict_keys in duplicate_values_dict.keys():
        if isinstance(duplicate_values_dict[dict_keys], list):
            duplicate_values_dict[dict_keys] = list(set(duplicate_values_dict[dict_keys]))
    for key, value in duplicate_values_dict.items():
        if isinstance(value, dict):
            remove_duplicate(value,dict_keys)

you can call this function using the dictionary and a key:

remove_duplicate(data,'last_name')

the key 'last_name' contains duplicate : ['GUYATT', 'GUYATT'] initially. After calling remove_duplicate(data,'last_name') you will find your 'data' contains only ['GUYATT'] for the key 'last_name'

print(data)

output:

{0: {key1 : {key2 :[value1,value2]}}}

from output you will find that the key 'last_name' contains['GUYATT']

edited Aug 02 '19 at 01:07

Gurpreet Singh Kaptan

21
3

answered Jul 29 '19 at 07:05

SM Abu Taher Asif

2,221
1
12
14

the code above is not working, It is just printing the value of the key, and not removing it. – Gurpreet Singh Kaptan Jul 29 '19 at 11:46
after running this code print your data variable and watch it will remove duplicate data from the given key – SM Abu Taher Asif Jul 29 '19 at 11:48
I have added the result, please check – SM Abu Taher Asif Jul 29 '19 at 11:51
Thanks, also can we make the parameter dict_keys as tuple, so that I can reuse it. – Gurpreet Singh Kaptan Jul 30 '19 at 20:14
as if I want to use, remove_duplicate(data, 'street_address' , 'last_name') can we make this fuction like that? – Gurpreet Singh Kaptan Jul 30 '19 at 21:52
If you want to call the function like remove_duplicate(data, 'street_address' , 'last_name') , you have to use variable number of argument When defining the function like: remove_duplicate(data, *dict_keys) . Inside the function you can loop through the dict_keys and get 'street_address' , 'last_name' – SM Abu Taher Asif Jul 31 '19 at 05:13
thanks bro, also since this function is not checking the value if its a list or not, it is creating one. how can I check it if this is a list, and then append it to a new dictionary, instead of using set() , as this will not maintain the order for later use. – Gurpreet Singh Kaptan Jul 31 '19 at 15:27
I have updated the code where I have checked for if dataType is list before using set – SM Abu Taher Asif Aug 01 '19 at 03:54

Remove duplicates from a nested dictionary

1 Answers1