How do I safely iterate over an optional list in Python?

Question

I have a function which compares a file path against two lists of folder paths. It returns true if the start of the path matches any folder from list A (include) but doesn't match an optional list B (exclude).

from typing import List, Optional

def match(file: str, include: List[str], exclude: Optional[List[str]]=None):
    return any(file.startswith(p) for p in include) and not any(
        file.startswith(p) for p in exclude
    )

The function works as expected if all parameter values are provided, but fails with a TypeError if no exclude folders are given.

# True
match(file="source/main.py", include=["source/", "output/"], exclude=["output/debug/"])

# False
match(file="output/debug/output.log", include=["source/", "output/"], exclude=["output/debug/"])

# Expected result - True
# Actual result - TypeError: 'NoneType' object is not iterable
match(file="output/debug/output.log", include=["source/", "output/"])

There was a PEP proposal to introduce null-aware operators that may have helped, but it has not been implemented as of now.

How can I safely iterate over an Optional list in Python without running into null errors?

Simply check if the list is none using an if-condition in the start. I think you could also try removing the `Optional` parameter and check. — The Myth, Feb 01 '23 at 18:53
Check if it's a list _before_ iterating over it? Maybe defaulting to an empty list, given the functionality in question. — jonrsharpe, Feb 01 '23 at 18:53
I was a big fan of using empty tuples for these kind of defaults, like `exclude=()`. But I guess the typing annotations don't jibe well with that. — wim, Feb 01 '23 at 18:56
You might be interested in [`more_itertools.always_iterable`](https://more-itertools.readthedocs.io/en/stable/api.html#more_itertools.always_iterable). — wim, Feb 01 '23 at 22:54

score 1 · Answer 1 · answered Feb 01 '23 at 19:01

Just use the "or []" technique to ensure "exclude" is only iterated when provided.

from typing import List, Optional

def match(file: str, include: List[str], exclude: Optional[List[str]]=None):
    return any(file.startswith(p) for p in include) and not any(
        file.startswith(p) for p in exclude or []
    )

>>> match('foo', ['f',])
True
>>> match('foo', ['f',], ['foo'])
False

score 1 · Answer 2 · answered Feb 01 '23 at 19:07

1

You don't seem to care if exclude is really a list, just that it's iterable. In that case, you can change the default to an immutable empty tuple if you change the type to Iterable[str].

from typing import Iterable


def match(file: str, include: Iterable[str], exclude:Iterable[str]=()):
    return (any(file.startswith(p) for p in include) 
            and all(not file.startswith(p) for p in exclude)

This also requires switching not any(... for p in exclude) to all(not ... for p in exclude) so that the test is vacuously true for the default value of exclude.

answered Feb 01 '23 at 19:07

chepner

497,756
71
530
681

How is this different than making it `=[]`? – The Myth Feb 01 '23 at 19:12
@TheMyth See: https://stackoverflow.com/questions/1132941/least-astonishment-and-the-mutable-default-argument – BeRT2me Feb 01 '23 at 19:12
As long as you trust your code to be checked with a tool like `mypy`, you could get away with `[]` because a function designed to work with *any* iterable type can't do anything to modify `exclude`, so `match` is "guaranteed" to keep an empty default list empty. (If you tried to do something like `exclude.append("foo")`, `mypy` would catch that error because not all `Iterable` types support `append`.) Using `()` is just an added layer of safety. – chepner Feb 01 '23 at 21:08

score 0 · Answer 3 · answered Feb 01 '23 at 19:42

Use an or condition to check if exclude is None before iterating through it.

from typing import List, Optional

def match(file: str, include: List[str], exclude: Optional[List[str]]=None):
    return any(file.startswith(p) for p in include) and (
        exclude is None or not any(file.startswith(p) for p in exclude)
    )

# True
match(file="output/debug/output.log", include=["source/", "output/"])

How do I safely iterate over an optional list in Python?

3 Answers3