Delete objects from a JSON file using a key obtained from a list of strings in Python

Question

I have a json in the following format:

{
  "features": [{
      "geometry": {
        "coordinates": [
          [
            [-12.345, 26.006],
            [-78.56, 24.944],
            [-76.44, 24.99],
            [-76.456, 26.567],
            [-78.345, 26.23456]
          ]
        ],

        "type": "Polygon"
      },

      "id": "Some_ID_01",

      "properties": {
        "parameters": "elevation"
      },
      "type": "Feature"
    },

    {
      "geometry": {
        "coordinates": [
          [
            [139.345, 39.2345],
            [139.23456, 37.3465],
            [141.678, 37.7896],
            [141.2345, 39.6543],
            [139.7856, 39.2345]
          ]
        ],
        "type": "Polygon"
      },
      "id": "Some_OtherID_01",
      "properties": {
        "parameters": "elevation"
      },
      "type": "Feature"
    }, {
      "geometry": {
        "coordinates": [
          [
            [143.8796, -30.243],
            [143.456, -32.764],
            [145.3452, -32.76],
            [145.134, -30.87],
            [143.123, -30.765]
          ]
        ],
        "type": "Polygon"
      },
      "id": "Some_ID_02",
      "properties": {
        "parameters": "elevation"
      },
      "type": "Feature"
    }
  ],
  "type": "FeatureCollection"
}

Im trying to remove any duplicates/older versions of the json object based on the id field (ie. the object with id=Some_ID_01 and id=Some_ID_02 are considered duplicates for my purposes).

So far I have manages to parse the json into python and create a list of all the IDs that require removal. I am stuck in actually using that list to delete/pop the objects from the json I parse in so I can rewrite the result to a new json file, not to mention it is far from optimized (my json file has some 20k objects in it)

This is my python code so far:

import json

json_file = open('features.json')
json_str = json_file.read()
json_data = json.loads(json_str)

dictionaryOfJsonId = {}
removalCounter = 0
keyToRemove = []
valueToRemoveFromList = []
IDList = []
removedSometing = 0

for values in json_data['features']:    #This loop converts the values in the json parse into a dict of only ID
    stringToSplit = values["id"]        #the id values from the json file
    IDList.append(stringToSplit)        #list with all the ID
    newKey = stringToSplit[:-2]         #takes the initial substring up to the last 2 spaces (version)
    newValue = stringToSplit[-2:]       #grabs the last two characters of the string

    if newKey in dictionaryOfJsonId:
        dictionaryOfJsonId[newKey].append(newValue)
    else:
        dictionaryOfJsonId[newKey] = [newValue]


for key in dictionaryOfJsonId:          #Remove entries that do not have duplicates
    if len(dictionaryOfJsonId[key])<2:
        valueToRemoveFromList.append(str(key + dictionaryOfJsonId[key][0]))
    else:
        valueToRemoveFromList.append(str(key +max(dictionaryOfJsonId[key])))


for string in valueToRemoveFromList:    #Remove all values that don't have duplicates from the List of ID
    IDList.remove(string)
    removalCounter+=1


for i in json_data['features']:
    for x in IDList:
        if i['id'] == x:
            json_data.pop(i)

The last for loop was my latest attempt at doing the deletion, but I get the error:

TypeError: unhashable type: 'dict'

`pop` expects an index, not an object, but that's irrelevant since it's a bad idea to modify an array that you're iterating over. Consider just using a list comprehension — Hamms, Jun 15 '17 at 21:48
something like `good_features = [i for i in json_data['feature'] if i['id'] not in IDList]` — Hamms, Jun 15 '17 at 21:49
Thanks @Hamms it worked! Post it below so I can mark as answer — Keki, Jun 16 '17 at 13:50

score 0 · Accepted Answer · answered Jun 16 '17 at 18:14

You're getting an error because pop expects an index, not an object.

However, that's somewhat irrelevant since it's a bad idea to modify a list that you're iterating over.

I'd consider just using a list comprehension; something like good_features = [i for i in json_data['feature'] if i['id'] not in IDList]

Delete objects from a JSON file using a key obtained from a list of strings in Python

1 Answers1

Linked