removing json items from array if value is duplicate python

Question

I am incredibly new to python.

I have an array full of json objects. Some of the json objects contain duplicated values. The array looks like this:

 [{"id":"1","name":"Paul","age":"21"},
  {"id":"2","name":"Peter","age":"22"},
  {"id":"3","name":"Paul","age":"23"}]

What I am trying to do is to remove an item if the name is the same as another json object, and leave the first one in the array.

So in this case I should be left with

  [{"id":"1"."name":"Paul","age":"21"},
  {"id":"2","name":"Peter","age":"22"}]

The code I currently have can be seen below and is largely based on this answer:

import json
ds = json.loads('python.json') #this file contains the json
unique_stuff = { each['name'] : each for each in ds }.values()

all_ids = [ each['name'] for each in ds ]
unique_stuff = [ ds[ all_ids.index(text) ] for text in set(texts) ]

print unique_stuff

I am not even sure that this line is working ds = json.loads('python.json') #this file contains the json as when I try and print ds nothing shows up in the console.

Please show us your non-working code so we can help you fix it. — PM 2Ring, Mar 28 '16 at 07:57

score 4 · Answer 1 · answered Mar 28 '16 at 08:05

4

You might have overdone in your approach. I might tend to rewrite the list as a dictionary with "name" as a key and then fetch the values

ds = [{"id":"1","name":"Paul","age":"21"},
  {"id":"2","name":"Peter","age":"22"},
  {"id":"3","name":"Paul","age":"23"}]

{elem["name"]:elem for elem in ds}.values()
Out[2]: 
[{'age': '23', 'id': '3', 'name': 'Paul'},
 {'age': '22', 'id': '2', 'name': 'Peter'}]

Off-course the items within the dictionary and the list may not be ordered, but I do not see much of a concern. If it is, let us know and we can think over it.

answered Mar 28 '16 at 08:05

Abhijit

62,056
18
131
204

thanks I will try that now. I am also trying to figure out how to get the json data into the python file I am writing my code in – Paul Fitzgerald Mar 28 '16 at 08:08
@PaulFitzgerald: I thought you already got that working? `ds = json.loads('python.json')` – Abhijit Mar 28 '16 at 08:09
I did too, but i dont think it is working as when i do `print ds` nothing shows in the terminal when I run the program, so I dont think its working. Can you see anything wrong with this code? or is it a python thing that wont show in the terminal? – Paul Fitzgerald Mar 28 '16 at 08:10
@PaulFitzgerald: `json.loads` expects its string argument to contain JSON data, not a file name. See pkacprzak's answer for the correct way to load JSON from a file. – PM 2Ring Mar 28 '16 at 08:17

score 3 · Answer 2 · answered Mar 28 '16 at 08:11

3

If you need to keep the first instance of "Paul" in your data a dictionary comprehension gives you the opposite result.

A simple solution could be as following

new = []
seen = set()
for record in old:
    name = record['name']
    if name not in seen:
        seen.add(name)
        new.append(record)
del seen

answered Mar 28 '16 at 08:11

gboffi

22,939
8
54
85

Thanks, nice solution – Paul Fitzgerald Mar 28 '16 at 08:15

score 1 · Accepted Answer · answered Mar 28 '16 at 08:11

First of all, your json snippet has invalid format - there are dot instead of commas separating some keys.

You can solve your problem using a dictionary with names as keys:

import json

with open('python.json') as fp:
    ds = json.load(fp) #this file contains the json

    mem = {}

    for record in ds:
        name = record["name"]
        if name not in mem:
            mem[name] = record

    print mem.values()

removing json items from array if value is duplicate python

3 Answers3

Linked