Is there a python-esque way to recursively check if a field is null?

Question

In other words, is there a prettier way to do the following?

if team is not None and team.captain is not None and team.captain.address is not None and team.captain.address.zipcode is not None:
  do_something()

where team is the instance of a class, and so are all the other inner fields to each other.

What changes if team is a dictionary instead? e.g. if team is not None and team['captain'] is not None... etc?

what is `team` - is it a dictionary object, something else? – rv.kvetch Oct 03 '22 at 15:54 — rv.kvetch, Oct 03 '22 at 15:54

score 9 · Accepted Answer · answered Oct 03 '22 at 15:57

I'd go with "it's easier to ask for forgiveness than permission":

try:
    has_captain_address_zipcode = (team.captain.address.zipcode is not None)
except AttributeError:  # presumably team/captain/address is None
    has_captain_address_zipcode = False

if has_captain_address_zipcode:
    # ...

The same stands for a nested dict:

try:
    has_captain_address_zipcode = (team["captain"]["address"]["zipcode"] is not None)
except (TypeError, KeyError):  # presumably team/captain/address is not a dict, or doesn't have the key
    has_captain_address_zipcode = False

if has_captain_address_zipcode:
    # ...

rv.kvetch · Answer 2 · 2022-10-03T18:57:44.030

Specifically, in regards to the question:

What changes if team is a dictionary instead? e.g. if team is not None and team['captain'] is not None... etc?

In case that teams is a dict object instead, I would suggest using something like dotwiz, a handy library I've created to enable dot-access for dictionary objects.

Use this alongside set_default_for_missing_keys(), in case a nested key might not exist:

from dotwiz import DotWiz, set_default_for_missing_keys

# only once
set_default_for_missing_keys(DotWiz())

team = {'captain': {'address': {'zipcode': 'test'}}}
team = DotWiz(team)

if team.captain.address.zipcode:
    print('hello')

team = DotWiz({'key': 'value'})

if team.captain.address.zipcode:
    print('world!')

Benchmarks

For in-depth benchmarks including comparison against other libraries, check out the Benchmarks section in the docs.

If anyone's curious, I put together a small performance comparison (create and __getitem__ times) with addict, in the case when a nested path doesn't exist.

For completeness I've also included a comparison with Dot, as suggested by @AKX in the comments; note that this implementation does not handle nested containers (i.e. within lists) currently. I've also went ahead and included the "simplest" nested dict approach by raising a KeyError, as also suggested.

from timeit import timeit

# pip install addict dotwiz
import addict
import dotwiz

# only once
dotwiz.set_default_for_missing_keys(dotwiz.DotWiz())


# implementation for Dot, as suggested by @AKX

class NotDot:
    __bool__ = lambda self: False
    __getattr__ = lambda self, name: self


NOT_DOT = NotDot()


class Dot:
    __slots__ = ('get', )

    def __init__(self, target):
        self.get = target.get

    def __getattr__(self, name, NOT_DOT=NOT_DOT):
        val = self.get(name, NOT_DOT)
        if type(val) is dict:
            return Dot(val)
        return val


# nested dict approach, as suggested by @AKX

def nested_dict_value(d):
    try:
        return d['captain']['address']['zipcode'] is not None
    except (TypeError, KeyError):  # presumably team/captain/address is not a dict, or doesn't have the key
        return False


if __name__ == '__main__':
    d1 = {'captain': {'address': {'zipcode': 'test'}}}
    d2 = {'key': 'value'}

    n = 100_000

    print('dict -> d1:   ', round(timeit('nested_dict_value(d1)', number=n, globals=globals()), 3))
    print('Dot -> d1:    ',
          round(timeit('team = Dot(d1); team.captain.address.zipcode', number=n, globals=globals()), 3))
    print('DotWiz -> d1: ',
          round(timeit('team = dotwiz.DotWiz(d1); team.captain.address.zipcode', number=n, globals=globals()), 3))
    print('addict -> d1: ',
          round(timeit('team = addict.Dict(d1); team.captain.address.zipcode', number=n, globals=globals()), 3))
    print()
    print('dict -> d2:    ', round(timeit('nested_dict_value(d2)', number=n, globals=globals()), 3))
    print('Dot -> d2:     ',
          round(timeit('team = Dot(d2); team.captain.address.zipcode', number=n, globals=globals()), 3))
    print('DotWiz -> d2:  ',
          round(timeit('team = dotwiz.DotWiz(d2); team.captain.address.zipcode', number=n, globals=globals()), 3))
    print('addict -> d2:  ',
          round(timeit('team = addict.Dict(d2); team.captain.address.zipcode', number=n, globals=globals()), 3))

    dot = Dot(d1)
    dw = dotwiz.DotWiz(d1)
    ad = addict.Dict(d1)

    print()
    print('Dot.get:     ', round(timeit('dot.captain.address.zipcode', number=n, globals=globals()), 3))
    print('DotWiz.get:  ', round(timeit('dw.captain.address.zipcode', number=n, globals=globals()), 3))
    print('addict.get:  ', round(timeit('ad.captain.address.zipcode', number=n, globals=globals()), 3))

    assert 'test' == dotwiz.DotWiz(d1).captain.address.zipcode == addict.Dict(d1).captain.address.zipcode

    assert not Dot(d2).captain.address.zipcode
    assert not dotwiz.DotWiz(d2).captain.address.zipcode
    assert not addict.Dict(d2).captain.address.zipcode

Results on my Mac M1:

dict -> d1:    0.009
Dot -> d1:     0.103
DotWiz -> d1:  0.09
addict -> d1:  0.375

dict -> d2:     0.012
Dot -> d2:      0.067
DotWiz -> d2:   0.087
addict -> d2:   0.381

Dot.get:      0.097
DotWiz.get:   0.007
addict.get:   0.074

An alternative to that library: [`addict`](https://github.com/mewwts/addict). — AKX, Oct 03 '22 at 16:04
Yep, good point. `addict` is another alternative worth considering as well :-) — rv.kvetch, Oct 03 '22 at 16:11
I've added a quick and dirty perf comparison against `addict` to my answer as well, as I was curious on the result. — rv.kvetch, Oct 03 '22 at 16:36
Nice – `dotwiz`'s global state with `set_default_for_missing_keys` gives me the heebie jeebies though. — AKX, Oct 03 '22 at 16:40
Yep, agreed - it's not an ideal solution, but at least I feel it beats out the alternative for now, i.e. passing a `default` value to the constructor method each time. — rv.kvetch, Oct 03 '22 at 16:43
By the way, it looks `dotwiz` is potentially slow (and/or memory-hungry) when the dict is very wide or deep, as it seems to do "dotwization" of the target object in advance, as opposed to on attribute access. — AKX, Oct 03 '22 at 17:11
@AKX totally right, but looking at the source code for `addict` it appears to more or less have the same approach for instantiation - iterate over the input dict and search for nested `dict` or `list` values. In the next major release, I'm planning on adding an option to skip iterating over nested types, which can be used to reduce memory usage a little. Other alternative libraries, such as the excellent [`scalpl`](https://pypi.org/project/scalpl/), should also be less "memory-hungry" when the dict is wide or deep, as it only looks up values on attribute or dot access. — rv.kvetch, Oct 03 '22 at 17:24
I came up with a small (read-only, on purpose) implementation which is much faster on wide/deep structures, and when it doesn't need to "dig in": https://gist.github.com/akx/11dddd8fd3027a410ff7a7d228f2cff1 Tradeoffs, tradeoffs... — AKX, Oct 03 '22 at 17:27
@AKX really cool, that seems pretty performant as well, and a lot less memory-hungry overall. I did find one minor drawback, it doesn't seem to handle deeply nested `dict` values, i.e. `{'captain': [{'address': {'zipcode': 'test'}}]}` - i.e. accessing like `team.captain[0].address.zipcode`. — rv.kvetch, Oct 03 '22 at 17:36
It doesn't currently do other containers than dicts at all, yep. — AKX, Oct 03 '22 at 17:47
For completeness, I've also included perf timings against the `Dot` implementation as mentioned. — rv.kvetch, Oct 03 '22 at 18:33

Yaakov Bressler · Answer 3 · 2022-10-03T17:10:27.960

You can create a list of dynamic lookup values and check if they're all not none:
(Though I'll add, it's a lot easier to ask for forgiveness, as mentioned above.)

# Create nested dynamic lookup of objects
my_objects = [
    team,
    'captain',
    'address',
    'zipcode'
]

def dynamic_recursive_lookup(obj, lookup_chain: list) -> list:
    """
    A bit complicated, but basically, safely dynamic lookup.
    Returns a list of objects, any of which may be None.
    """
    result = [obj]
    prev = obj
    for x in lookup_chain[1:]:
        curr = getattr(prev, x, None)
        result.append(curr)
        prev = curr

    return result


# Get list of dynamic objects
my_dynamic_objects = dynamic_recursive_lookup(team, my_objects)

# Now check if all are not none:
if all(x is not None for x in my_dynamic_objects):
    doSomething()

Another alternative is a default dict, though that would require converting your object to a dictionary, which may not be possible?

Is there a python-esque way to recursively check if a field is null?

3 Answers3

Benchmarks