Removing nulls and empty objects of mixed data types from a dictionary with a nested structure

Question

Off the back of a similar question I asked, how would one go about cleaning a dictionary containing a variety of datatypes: nulls, empty lists, empty dicts etc, at varying levels of nesting E.g.

{
  "key":"value",
  "key1": {},
  "key2": [],
  "key3": True,
  "key4": False,
  "key5": None,
  "key6": [1,2,3],
  "key7": {
    "subkey": "subvalue"
  },
  "key8": {
    "subdict": {
      "subdictkey": "subdictvalue", 
      "subdictkey1": {},
      "subdictkey2": [],
      "subdictkey3": None
    }
  }
}

Becomes:

{
  "key":"value",
  "key3": True,
  "key4": False,
  "key6": [1,2,3],
  "key7": {
    "subkey": "subvalue"
  },
  "key8": {
    "subdict": {
      "subdictkey": "subdictvalue"
    }
  }
}

The solution should be good for n levels of nesting (not just 1 level). Obviously I want to avoid nested loops (particularly as n could equal 3 or 4), is the only solution flattening the structure? Is there a more elegant way of going about it?

Edit: Building on @Ch3steR answer and accounting for the issue I encountered with a list containing a null, this is the final working function:

def recur(n_dict,new_d={}):
    global counter
    for key,val in n_dict.items():
        if val or isinstance(val,bool) or (isinstance(val,list) and any(elem is not None for elem in val)):
                if (isinstance(val,list) and any(elem is None for elem in val)):
                   counter=counter+1
                else:
                    new_d={**new_d,**{key:val}}
                    if isinstance(val,dict):
                        new_d[key]=recur(val)

    return new_d

You asked the same question yesterday. https://stackoverflow.com/questions/60381477/removing-nulls-and-empty-objects-of-mixed-data-types-from-a-dictionary Just apply the same logic using recursion. — Ch3steR, Feb 25 '20 at 12:24
Does this answer your question? [Removing nulls and empty objects of mixed data types from a dictionary](https://stackoverflow.com/questions/60381477/removing-nulls-and-empty-objects-of-mixed-data-types-from-a-dictionary) — Ch3steR, Feb 25 '20 at 12:25
I thought the addition of the structural component was a substantial enough change that it warranted another question, should I just update the previous one? — AJG, Feb 25 '20 at 12:29
It would be better to incorporate the solutions from the previous question into answering your current question. When you have n-level nested dictionary use ***Recursion***. — Ch3steR, Feb 25 '20 at 12:32
The answer you accepted in that question is not *pythonic*. And to check if `list`,`tuple`,`dict` is empty use `if data_structure` not `if len(data_structure)==0` like proposed in the accepted answer of your previous. — Ch3steR, Feb 25 '20 at 12:34
I see, honestly, I accepted it as it was the first one that worked, and yours was below that. I'm happy to have a solution that is native with more finesse though, so I'll employ yours instead. I now need to go away and read up on recursive functions I guess. Thanks — AJG, Feb 25 '20 at 12:41

Ch3steR · Accepted Answer · 2020-02-25T13:37:26.707

2

You can use Recursion when you are dealing with an arbitrarily nested dictionary.

Try this.

def recur(n_dict,new_d={}):
    for key,val in n_dict.items():
        if val or isinstance(val,bool):
            new_d={**new_d,**{key:val}}
            if isinstance(val,dict):
                new_d[key]=recur(val)
    return new_d

a={
  "key":"value",
  "key1": {},
  "key2": [],
  "key3": True,
  "key4": False,
  "key5": None,
  "key6": [1,2,3],
  "key7": {
    "subkey": "subvalue"
  },
  "key8": {
    "subdict": {
      "subdictkey": "subdictvalue", 
      "subdictkey1": {},
      "subdictkey2": [],
      "subdictkey3": None
    }
  }
}

print(recur(a))
{'key': 'value',
 'key3': True,
 'key4': False,
 'key6': [1, 2, 3],
 'key7': {'subkey': 'subvalue'},
 'key8': {'subdict': {'subdictkey': 'subdictvalue'}}}

recur(n_dict,new_d={}) uses mutable default argument.

Note:

Never mutate new_d in-place or you will encounter this problem.

One of the way to check if your default argument is changed is use __defaults__

>>>recur(a)
>>>recur.__defaults__
({},) 
>>>recur(a)
>>>recur.__defaults__
({},)

edited Feb 25 '20 at 13:37

answered Feb 25 '20 at 13:31

Ch3steR

20,090
4
28
58

This is great thanks. Would you mind explaining what the 4th line of your function is doing? Also how would one add in an extra condition to remove lists that contain a single null e.g. `[None]`. I tried adding an extra condition on the end of line 3 of your function `or (isinstance(val,list) and any(elem is not None for elem in val)` with no luck – AJG Feb 25 '20 at 16:21
@AJG You need to tweak it a little bit. Now in the present function you recur when `val` is *dictionary*. Now you need to check if the `val` is an *Iterable* but not a string. Implement this into the above function. If fail to implement this comment below. I'll add the answer again. – Ch3steR Feb 25 '20 at 16:28
@AJG It's really hard to read code from comments. Did it work? – Ch3steR Feb 25 '20 at 17:32
I updated the question with the final working code, if you have any final pointers? – AJG Feb 25 '20 at 17:33

Removing nulls and empty objects of mixed data types from a dictionary with a nested structure

1 Answers1