How to remove duplicates from list of dictionaries in python

Question

I have 2 list of dictionaries with the same values. I want to remove 1 value.

This is how it looks:

[
{
  "activity_name": "Wait Time",
  "task_detail": [
    {
      "created_date": "2021-04-29 10:00:25.695254",
      "row_id": ""
    }
  ]
},
{
  "activity_name": "Wait Time",
  "task_detail": [
    {
      "created_date": "2021-04-29 10:28:42.131017",
      "row_id": ""
    }
  ]
}
]

Here is my code:

result=[]
for gc_item in gc_response['Items']:
    for sch_act in gc_item['scheduled_activity']:
        result.append(sch_act)

How to remove duplicate value for this? Under Scheduled_activity only i have the value(Activity name and task_detail)

I need to remove 1 of the duplicate value. The output should be:

{
  "activity_name": "Wait Time",
  "task_detail": [
    {
      "created_date": "2021-04-29 10:28:42.131017",
      "row_id": ""
    }
  ]
}

Please edit your question to include an example of correct output. I see that you have provided an example of input. However, we really need to see an example of correct output. What is the computer program supposed to produce at the very end? — Toothpick Anemone, May 06 '21 at 03:26
"I need to remove 1 of the duplicate value". What is the rule that tells you which one to remove? What will you do if there is more than one duplicate? What is the rule that tells you whether or not one value duplicates another (what exactly needs to match)? — Karl Knechtel, May 06 '21 at 03:36

score 0 · Answer 1 · answered May 06 '21 at 03:32

You can only add if the activity_name doesn't already exist in list

result=[]
acitivity_names_added = []
for sch_act in gc_item['scheduled_activity']:
    if(sch_act['activity_name'] not in acitivity_names_added):
        result.append(sch_act) 
        acitivity_names_added.append(sch_act['activity_name'])

If you want to keep the last one instead of the first one then iterate through this loop in reverse order

result=[]
acitivity_names_added = []
for sch_act in gc_item['scheduled_activity'][::-1]:
    if(sch_act['activity_name'] not in acitivity_names_added):
        result.append(sch_act) 
        acitivity_names_added.append(sch_act['activity_name'])

score 0 · Answer 2 · answered May 06 '21 at 03:43

This is similar to Remove duplicate dict in list in Python

In your case, task_detail seems to be more complicated, so the easy tuple-method like [dict(t) for t in {tuple(d.items()) for d in l}] may not work, but you can try the json method in the same thread:

import json

result = []
for gc_item in gc_response['Items']:
    set_of_jsons = {json.dumps(d, sort_keys=True) for d in gc_item['scheduled_activity']}
    result += [json.loads(t) for t in set_of_jsons]

It basically turns list/dict to string so that you could remove duplicates by set.

How to remove duplicates from list of dictionaries in python

2 Answers2