2

Consider this contrived* example:

def count(one, two, three):
    print one
    print two
    print three

Three shall be the number thou shalt count, and the number of the counting shall be three.

>>> x = [1, 2, 3]
>>> count(*map(int, x), three=x.pop())
1
2
3

Four shalt thou not count,

>>> x = [1, 2, 3, 4]
>>> count(*map(int, x), three=x.pop())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: count() got multiple values for keyword argument 'three'

neither count thou two, excepting that thou then proceed to three.

>>> x = [1, 2]
>>> count(*map(int, x), three=x.pop())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: count() takes exactly 3 arguments (2 given)

Five is right out.

>>> x = [1, 2, 3, 4, 5]
>>> count(*map(int, x), three=x.pop())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: count() takes exactly 3 arguments (5 given)

After having read this question, I would actually have thought that x = [1, 2] is the only one that works, because

  • first, map(int, x) would be evaluated, one set to 1 and two set to 2
  • then, x still [1, 2], x.pop() would be evaluated and three set to 2, too.

My expectation for x = [1, 2, 3] was to get the error that I actually saw for x = [1, 2, 3, 4].

What is going on here? Why are the arguments seemingly not evaluated from left to right? Are keyword arguments evaluated first?


*actually my real code corresponds to x = [1, 2, 3], which works, but I wasn't sure it was safe, and after reading the other question I thought it shouldn't actually work.

I'm using Python 2.7, if that matters.

Community
  • 1
  • 1
mkrieger1
  • 19,194
  • 5
  • 54
  • 65
  • You pop an element off so you are left with two args in the first example – Padraic Cunningham Oct 20 '15 at 19:18
  • But why do I `pop` before I `map`? – mkrieger1 Oct 20 '15 at 19:19
  • Because reasons. Honestly there's no obvious expected behavior here. If any of these worked I would consider it entirely luck and wouldn't rely on it. Write clearly and avoid mutation where possible otherwise it's very difficult to reason about what should happen. – Chad S. Oct 20 '15 at 19:29
  • 1
    The statement and the example following it below the "CPython implementation detail" seems relevant here: [Python Language Reference » Expressions » Calls](https://docs.python.org/3/reference/expressions.html#calls): "although the `*expression` syntax may appear *after* some keyword arguments, it is processed *before* the keyword arguments" – Lukas Graf Oct 20 '15 at 20:10

1 Answers1

2

Python 2.7

If we look at the CPython source related to creating the AST(ast_for_call) for a function call the order of argument evaluation turns out to be:

return Call(func, args, keywords, vararg, kwarg, func->lineno,
                func->col_offset, c->c_arena);

ie. args --> keywords --> vararg --> kwarg

So, in your case the keyword argument is evaluated first and then the star based expression(vararg) is evaluated.

Byte code:

>>> dis.dis(lambda: func(1, 2, *('k', 'j', 'l'), z=1, y =2, three=x.pop(), **{kwarg:1}))
  1           0 LOAD_GLOBAL              0 (func)
              3 LOAD_CONST               1 (1)         # arg
              6 LOAD_CONST               2 (2)         # arg
              9 LOAD_CONST               3 ('z')       # keyword
             12 LOAD_CONST               1 (1)
             15 LOAD_CONST               4 ('y')       # keyword
             18 LOAD_CONST               2 (2)
             21 LOAD_CONST               5 ('three')   # keyword
             24 LOAD_GLOBAL              1 (x)
             27 LOAD_ATTR                2 (pop)
             30 CALL_FUNCTION            0
             33 LOAD_CONST               9 (('k', 'j', 'l')) #vararg
             36 BUILD_MAP                1
             39 LOAD_CONST               1 (1)
             42 LOAD_GLOBAL              3 (kwarg)     #kwarg
             45 STORE_MAP
             46 CALL_FUNCTION_V

Hence in your case the pop() call will happen first followed by the varargs evaluation.

So, if three is a part of kwargs then we will get an error with map:

>>> x = [1, 2, 3]
>>> count(*map(float, x), **{'three': x.pop()})
Traceback (most recent call last):
  File "<ipython-input-133-e8831565af13>", line 1, in <module>
    count(*map(float, x), **{'three': x.pop()})
TypeError: count() got multiple values for keyword argument 'three'

It will work if we do it *lazily:

>>> x = [1, 2, 3]
>>> count(*(float(y) for y in x), **{'three': x.pop()})
1.0, 2.0, 3

*The reason why generator works and map or list comprehension fails is explained at the end.


Python 3.5

The ast_for_call function here only maintains two lists: args and keywords.

Here the varargs are inserted into the args list and kwargs go to the keywords list. So, in the end the call looks like:

return Call(func, args, keywords, func->lineno, func->col_offset, c->c_arena);

Byte code:

>>> dis.dis(lambda: func(1, 2, *('k', 'j', 'l'), z=1, y =2, three=x.pop(), **{kwarg:1}))
  1           0 LOAD_GLOBAL              0 (func)
              3 LOAD_CONST               1 (1)
              6 LOAD_CONST               2 (2)
              9 LOAD_CONST               9 (('k', 'j', 'l'))
             12 LOAD_CONST               6 ('z')
             15 LOAD_CONST               1 (1)
             18 LOAD_CONST               7 ('y')
             21 LOAD_CONST               2 (2)
             24 LOAD_CONST               8 ('three')
             27 LOAD_GLOBAL              1 (x)
             30 LOAD_ATTR                2 (pop)
             33 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
             36 LOAD_GLOBAL              3 (kwarg)
             39 LOAD_CONST               1 (1)
             42 BUILD_MAP                1
             45 CALL_FUNCTION_VAR_KW   770 (2 positional, 3 keyword pair)
             48 RETURN_VALUE

Now things can get a little exciting if the expression yielding the varargs is lazy:

>> def count(one, two, three):
        print (one, two, three)
...
>>> x = [1, 2, 3]
>>> count(*map(float, x), three=x.pop())  # map is lazy in Python 3
1.0 2.0 3
>>> x = [1, 2, 3]
>>> count(*[float(y) for y in x], three=x.pop())
Traceback (most recent call last):
  File "<ipython-input-25-b7ef8034ef4e>", line 1, in <module>
    count(*[float(y) for y in x], three=x.pop())
TypeError: count() got multiple values for argument 'three'

Byte code:

>>> dis.dis(lambda: count(*map(float, x), three=x.pop()))
  1           0 LOAD_GLOBAL              0 (count)
              3 LOAD_GLOBAL              1 (map)
              6 LOAD_GLOBAL              2 (float)
              9 LOAD_GLOBAL              3 (x)
             12 CALL_FUNCTION            2 (2 positional, 0 keyword pair)
             15 LOAD_CONST               1 ('three')
             18 LOAD_GLOBAL              3 (x)
             21 LOAD_ATTR                4 (pop)
             24 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
             27 CALL_FUNCTION_VAR      256 (0 positional, 1 keyword pair)
             30 RETURN_VALUE
>>> dis.dis(lambda: count(*[float(y) for y in x], three=x.pop()))
  1           0 LOAD_GLOBAL              0 (count)
              3 LOAD_CONST               1 (<code object <listcomp> at 0x103b63930, file "<ipython-input-28-1cc782164f20>", line 1>)
              6 LOAD_CONST               2 ('<lambda>.<locals>.<listcomp>')
              9 MAKE_FUNCTION            0
             12 LOAD_GLOBAL              1 (x)
             15 GET_ITER
             16 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             19 LOAD_CONST               3 ('three')
             22 LOAD_GLOBAL              1 (x)
             25 LOAD_ATTR                2 (pop)
             28 CALL_FUNCTION            0 (0 positional, 0 keyword pair)
             31 CALL_FUNCTION_VAR      256 (0 positional, 1 keyword pair)
             34 RETURN_VALUE

The lazy call works because unpacking(aka actual evaluation of the generator) doesn't happen until the function is actually called, hence in this case pop() call will remove the 3 first and then later on map will only pass 1, 2.

But, in the case of list comprehension the list object already contains 3 items and then even though pop() removed 3 later on we are still passing two values for the third argument.

Community
  • 1
  • 1
Ashwini Chaudhary
  • 244,495
  • 58
  • 464
  • 504
  • Thanks for the detailed insight! So to summarize: It happens to work in CPython 2.7, but not in general in 3.5, except when using `map`; but it's an implementation detail and cannot be relied on? – mkrieger1 Oct 21 '15 at 09:27
  • @mkrieger1 It's not an implementation detail but a version detail, Python 2 and Python 3 are handling things differently. But across different implementation the output should match, for example I tested on PyPy 2.7.9 and it works same as CPython 2.7.10. Things have changed between Python 3.4 and 3.5 as well, for example now something like `count(*(1,), *(2, 3))` is valid(because of [PEP-448](https://docs.python.org/3/whatsnew/3.5.html#whatsnew-pep-448)) in Python 3.5 but not in older versions of Python 3.x. – Ashwini Chaudhary Oct 21 '15 at 10:47