Check key existence in nested dictionaries

Question

I am working with API calls and thus Python dictionaries.

However, for the same request, I don't always key the same keys, and I'd like to know when I can call a key without an Exception...

Let's say I have

test = {'a':{'b':{'c':{'d':'e'}}}}

Sometimes key d will exist, sometimes it won't. Sometimes c won't even exist.

I'd like to, somehow, check if test['a']['b']['c']['d'] exists, in one line.

What I've tried so far:

• Using test.get('a', {}).get('b', {}).get('c', {}).get('d', {}). Works fine but it's a mess, sometimes I have 5-6 nested dictionaries with really long names...

• Using try/except block which is nice, but usually if test['a']['b']['c']['d'] does not exist, I will try calling test['a']['b']['e']['f'] to check if that one exists, thus I would need to add a try/catch for every of my if statements, as if I am not wrong, if an exception is catch, try block is not executed anymore.

I was maybe trying to look to a somekind of reflexive way to do it, calling a function with the name of my "object" as a string, that would check if every key exists, and if it does, return the object itself.

Any thoughts?

The usage behind it would be, omitting useless case, and assuming sometimes the info is in test['a']['b']['c']['d'], sometimes in test['a']['b']['f'] :

if test['a']['b']['c']['d'] **exists**:
    do sthg with the value of test['a']['b']['c']['d']
elif test['a']['b']['f'] **exists**:
    do sthg else with the value of test['a']['b']['f']
else:
    do sthg different

If I put a try/except there, won't the first exception stop the execution and don't let me execute the elif?

Moreover, I really like the way of calling test['a']['b']['c']['d'] better than giving a list of keys. In fact, I want it to be as transparent as possible for me and for the people who will read/use my code.

Answer 1

You could write a recursive function to check:

def f(d, keys):
    if not keys:
        return True
    return keys[0] in d and f(d[keys[0]], keys[1:])

If the function returns True, the keys exist:

In [10]: f(test,"abcd")
Out[10]: True

In [11]: f(test,"abce")
Out[11]: False

If you want to test multiple key combinations:

for keys in ("abce","abcr","abcd"):
    if f(test,keys):
        print(keys)
        break
abcd

To return the value it is pretty simple:

def f(d, keys):
    if len(keys) == 1:
         return d[keys[0]] if keys[0] in d else False
    return keys[0] in d and f(d[keys[0]], keys[1:])

print(f(test,"abcd"))
e

You can test again for multiple key combinations:

def test_keys(keys):
    for keys in keys:
        val = f(test,keys)
        if val:
            return val
    return False


print(test_keys(("abce","abcr","abc")))

You can also write the function iteratively:

def f(d, keys):
    obj = object
    for k in keys:
        d = d.get(k, obj)
        if d is obj:
            return False
    return d

print(f(test,"abcd"))
e

If you want to run a condition based on the return values:

def f(d, keys):
    obj = object
    for k in keys:
        d = d.get(k, obj)
        if d is obj:
            return False
    return d

from operator import mul

my_actions = {"c": mul(2, 2), "d": lambda: mul(3, 3), "e": lambda: mul(3, 3)}

for st in ("abce", "abcd", "abcf"):
    val = f(test, st)
    if val:
        print(my_actions[val]())
9

Just test the key combo in the same order you would with your if/elif's etc..

Answer 2

It's not exactly what you want because it doesn't check existence, but here's a one-liner similar to the dict.get method:

In [1]: test = {'a':{'b':{'c':{'d':'e'}}}}
In [2]: keys = 'abcd' # or ['a', 'b', 'c', 'd']

In [3]: reduce(lambda d, k: d.get(k) if d else None, keys, test)
Out[3]: 'e'

In [4]: keys = 'abcf'

In [5]: reduce(lambda d, k: d.get(k) if d else None, keys, test)

Unfortunately it's not very efficient because it doesn't stop as soon as one of the keys is missing.

Answer 3

I would create a recursive function.

def get_key(d, *args):
    if not args:
        return None
    val = d.get(args[0], None)

    if len(args) == 1:
        return val
    if isinstance(val, dict):
        return get_key(val, *args[1:])

Answer 4

You could nest try blocks to handle exceptions from both kinds of missing key. Relying on try blocks fits the EAFP (easier to ask forgiveness than permission) philosophy of Python more than the LBYL (look before you leap) pattern of testing existence before use. In a multithreaded program, it also helps avoid unexpected TOCTTOU (time of check to time of use) behavior caused by another thread modifying the test dictionary between the test for existence and the use of the value.

try:
    value_abcd = test['a']['b']['c']['d']
except KeyError:
    try:
        value_abf = test['a']['b']['f']
    except KeyError:
        print("do something different")
    else:
        print("value_abf is", value_abf)
else:
    print("value_abcd is", value_abcd)

It has since come to my attention that you have far more than two keys. Nesting try blocks with this many keys would create an arrowhead anti-pattern. So instead, you can try the following construction to handle all keys at the same indent level, so long as the access occurs in a function or a for loop so that it can return or continue. If not, extract a method.

try:
    value_abcd = test['a']['b']['c']['d']
except KeyError:
    pass
else:
    print("value_abcd is", value_abcd)
    return  # or continue if doing this in a loop

try:
    value_abf = test['a']['b']['f']
except KeyError:
    pass
else:
    print("value_abf is", value_abf)
    return

print("do something different")

Answer 5

If you are working with JSON, you can write a simple class to use with the dict as it is imported.

Given the following bit of JSON:

>>> js='{"a": {"b": {"c": {"d": "e"}}}}'

It would usually be decoded into a Python dict if it is comprised of object pairs:

>>> import json
>>> json.loads(js)
{u'a': {u'b': {u'c': {u'd': u'e'}}}}

As a normal Python dict, it is subject to KeyError with missing keys. You can use the __missing__ hook to override KeyErrors and achieve your original structure:

class Mdict(dict):
    def __missing__(self, key):
        return False

Now test that:

>>> md=Mdict({'a':Mdict({'b':Mdict({'c':Mdict({'d':'e'})})})})
>>> if md['a']['b']['d']:
...    print md['a']['b']['d']
... elif md['a']['b']['c']:
...    print 'elif', md['a']['b']['c']  
... 
elif {'d': 'e'}

Each level of the dict needs to be an Mdict vs a normal Python dict. If you are working with JSON, however, this is really easy to achieve. Just apply object_pairs_hook as you decode the JSON:

>>> js
'{"a": {"b": {"c": {"d": "e"}}}}'
>>> md=json.loads(js, object_pairs_hook=Mdict)

And that applies the class Mdict instead the default Python dict as the JSON is decoded.

>>> md
{u'a': {u'b': {u'c': {u'd': u'e'}}}}
>>> md['a']
{u'b': {u'c': {u'd': u'e'}}}
>>> md['a']['c']
False

The rest of the example here remains the same.

Answer 6

Another one for completeness, but checks only for one key recursively:

def hasKey(d, key):
    found = False
    if isinstance(d, dict):
        for k in d:
            found = True if k == key else found or hasKey(d[k], key)
    if isinstance(d, list):
        for i in d:
            found = found or hasKey(i, key)
    return found

Checks if key exists as a key in the nested dict map.