How can I avoid issues caused by Python's early-bound default parameters (e.g. mutable default arguments "remembering" old data)?

Question

Sometimes it seems natural to have a default parameter which is an empty list. However, Python produces unexpected behavior in these situations.

For example, consider this function:

def my_func(working_list=[]):
    working_list.append("a")
    print(working_list)

The first time it is called, the default will work, but calls after that will update the existing list (with one "a" each call) and print the updated version.

How can I fix the function so that, if it is called repeatedly without an explicit argument, a new empty list is used each time?

Answer 1

def my_func(working_list=None):
    if working_list is None: 
        working_list = []

    # alternative:
    # working_list = [] if working_list is None else working_list

    working_list.append("a")
    print(working_list)

The docs say you should use None as the default and explicitly test for it in the body of the function.

Answer 2

Other answers have already already provided the direct solutions as asked for, however, since this is a very common pitfall for new Python programmers, it's worth adding the explanation of why Python behaves this way, which is nicely summarized in The Hitchhikers Guide to Python under Mutable Default Arguments:

Python's default arguments are evaluated once when the function is defined, not each time the function is called (like it is in say, Ruby). This means that if you use a mutable default argument and mutate it, you will and have mutated that object for all future calls to the function as well.

Answer 3

Not that it matters in this case, but you can use object identity to test for None:

if working_list is None: working_list = []

You could also take advantage of how the boolean operator or is defined in python:

working_list = working_list or []

Though this will behave unexpectedly if the caller gives you an empty list (which counts as false) as working_list and expects your function to modify the list he gave it.

Answer 4

If the intent of the function is to modify the parameter passed as working_list, see HenryR's answer (=None, check for None inside).

But if you didn't intend to mutate the argument, just use it as starting point for a list, you can simply copy it:

def myFunc(starting_list = []):
    starting_list = list(starting_list)
    starting_list.append("a")
    print starting_list

(or in this simple case just print starting_list + ["a"] but I guess that was just a toy example)

In general, mutating your arguments is bad style in Python. The only functions that are fully expected to mutate an object are methods of the object. It's even rarer to mutate an optional argument — is a side effect that happens only in some calls really the best interface?

• If you do it from the C habit of "output arguments", that's completely unnecessary - you can always return multiple values as a tuple.

• If you do this to efficiently build a long list of results without building intermediate lists, consider writing it as a generator and using result_list.extend(myFunc()) when you are calling it. This way your calling conventions remains very clean.

One pattern where mutating an optional arg is frequently done is a hidden "memo" arg in recursive functions:

def depth_first_walk_graph(graph, node, _visited=None):
    if _visited is None:
        _visited = set()  # create memo once in top-level call

    if node in _visited:
        return
    _visited.add(node)
    for neighbour in graph[node]:
        depth_first_walk_graph(graph, neighbour, _visited)

Answer 5

I might be off-topic, but remember that if you just want to pass a variable number of arguments, the pythonic way is to pass a tuple *args or a dictionary **kargs. These are optional and are better than the syntax myFunc([1, 2, 3]).

If you want to pass a tuple:

def myFunc(arg1, *args):
  print args
  w = []
  w += args
  print w
>>>myFunc(1, 2, 3, 4, 5, 6, 7)
(2, 3, 4, 5, 6, 7)
[2, 3, 4, 5, 6, 7]

If you want to pass a dictionary:

def myFunc(arg1, **kargs):
   print kargs
>>>myFunc(1, option1=2, option2=3)
{'option2' : 2, 'option1' : 3}

Answer 6

Quote from https://docs.python.org/3/reference/compound_stmts.html#function-definitions

Default parameter values are evaluated from left to right when the function definition is executed. This means that the expression is evaluated once, when the function is defined, and that the same “pre-computed” value is used for each call. This is especially important to understand when a default parameter is a mutable object, such as a list or a dictionary: if the function modifies the object (e.g. by appending an item to a list), the default value is in effect modified. This is generally not what was intended. A way around this is to use None as the default, and explicitly test for it in the body of the function, e.g.:

def whats_on_the_telly(penguin=None):
    if penguin is None:
        penguin = []
    penguin.append("property of the zoo")
    return penguin

Answer 7

Perhaps the simplest thing of all is to just create a copy of the list or tuple within the script. This avoids the need for checking. For example,

    def my_funct(params, lst = []):
        liste = lst.copy()
         . . 

Answer 8

Recap

Python evaluates default values for arguments/parameters ahead of time; they are "early-bound". This can cause problems in a few different ways. For example:

>>> import datetime, time
>>> def what_time_is_it(dt=datetime.datetime.now()): # chosen ahead of time!
...     return f'It is now {dt.strftime("%H:%M:%S")}.'
... 
>>> 
>>> first = what_time_is_it()
>>> time.sleep(10) # Even if time elapses...
>>> what_time_is_it() == first # the reported time is the same!
True

The most common way the problem manifests, however, is when the argument to the function is mutable (for example, a list), and gets mutated within the function's code. When this happens, changes will be "remembered", and thus "seen" on subsequent calls:

>>> def append_one_and_return(a_list=[]):
...     a_list.append(1)
...     return a_list
... 
>>> 
>>> append_one_and_return()
[1]
>>> append_one_and_return()
[1, 1]
>>> append_one_and_return()
[1, 1, 1]

Because a_list was created ahead of time, every call to the function that uses the default value will use the same list object, which gets modified on each call, appending another 1 value.

This is a conscious design decision that can be exploited in some circumstances - although there are often better ways to solve those other problems. (Consider using functools.cache or functools.lru_cache for memoization, and functools.partial to bind function arguments.)

This also implies that methods of an instance cannot use an attribute of the instance as a default: at the time that the default value is determined, self is not in scope, and the instance does not exist anyway:

>>> class Example:
...     def function(self, arg=self):
...         pass
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in Example
NameError: name 'self' is not defined

(The class Example also doesn't exist yet, and the name Example is also not in scope; therefore, class attributes will also not work here, even if we don't care about the mutability issue.)

Solutions

Using None as a sentinel value

The standard, generally-considered-idiomatic approach is to use None as the default value, and explicitly check for this value and replace it in the function's logic. Thus:

>>> def append_one_and_return_fixed(a_list=None):
...     if a_list is None:
...         a_list = []
...     a_list.append(1)
...     return a_list
... 
>>> append_one_and_return_fixed([2]) # it works consistently with an argument
[2, 1]
>>> append_one_and_return_fixed([2])
[2, 1]
>>> append_one_and_return_fixed() # and also without an argument
[1]
>>> append_one_and_return_fixed()
[1]

This works because the code a_list = [] runs (if needed) when the function is called, not ahead of time - thus, it creates a new empty list every time. Therefore, this approach can also solve the datetime.now() issue. It does mean that the function can't use a None value for other purposes; however, this should not cause a problem in ordinary code.

Simply avoiding mutable defaults

If it is not necessary to modify the argument in order to implement the function's logic, because of the principle of command-query separation, it would be better to just not do that.

By this argument, append_one_and_return is poorly designed to begin with: since the purpose is to display some modified version of the input, it should not also actually modify the caller's variable, but instead just create a new object for display purposes. This allows for using an immutable object, such as a tuple, for the default value. Thus:

def with_appended_one(a_sequence=()):
    return [*a_sequence, 1]

This way will avoid modifying the input even when that input is explicitly provided:

>>> x = [1]
>>> with_appended_one(x)
[1, 1]
>>> x # not modified!
[1]

It works fine without an argument, even repeatedly:

>>> with_appended_one()
[1]
>>> with_appended_one()
[1]

And it has gained some flexibility:

>>> with_appended_one('example') # a string is a sequence of its characters.
['e', 'x', 'a', 'm', 'p', 'l', 'e', 1]

Possible new syntax in 3.12

There are plans to introduce new syntax in Python 3.12, documented in PEP 671, that would allow an explicit request for late binding of default values rather than early binding. The syntax would most likely look like:

def append_and_show_future(a_list=>None): # note => instead of =
    a_list.append(1)
    print(a_list)

However, as of this writing, the proposal is not formally accepted and not on the list of incoming changes.

Answer 9

There have already been good and correct answers provided. I just wanted to give another syntax to write what you want to do which I find more beautiful when you for instance want to create a class with default empty lists:

class Node(object):
    def __init__(self, _id, val, parents=None, children=None):
        self.id = _id
        self.val = val
        self.parents = parents if parents is not None else []
        self.children = children if children is not None else []

This snippet makes use of the if else operator syntax. I like it especially because it's a neat little one-liner without colons, etc. involved and it nearly reads like a normal English sentence. :)

In your case you could write

def myFunc(working_list=None):
    working_list = [] if working_list is None else working_list
    working_list.append("a")
    print working_list

Answer 10

I took the UCSC extension class Python for programmer

Which is true of: def Fn(data = []):

a) is a good idea so that your data lists start empty with every call.

b) is a good idea so that all calls to the function that do not provide any arguments on the call will get the empty list as data.

c) is a reasonable idea as long as your data is a list of strings.

d) is a bad idea because the default [] will accumulate data and the default [] will change with subsequent calls.

Answer:

d) is a bad idea because the default [] will accumulate data and the default [] will change with subsequent calls.