1

To start with, my question here is about the semantics and the logic behind why the Python language was designed like this in the case of chained decorators. Please notice the nuance how this is different from the question

How decorators chaining work?

Link: How decorators chaining work? It seems quite a number of other users had the same doubts, about the call order of chained Python decorators. It is not like I can't add a __call__ and see the order for myself. I get this, my point is, why was it designed to start from the bottom, when it comes to chained Python decorators?

E.g.

def first_func(func):
    def inner():
        x = func()
        return x * x
    return inner

def second_func(func):
    def inner():
        x = func()
        return 2 * x
    return inner

@first_func
@second_func
def num():
    return 10

print(num())
apingaway
  • 33
  • 1
  • 1
  • 17
  • 2
    What are you defining as "backwards"? When calling the decorated function, `first_func` will receive the actual call, and then call `second_func`, which then calls `num`. Exactly in the order as written. The first decorator becomes the outermost wrapper. – deceze Dec 09 '22 at 09:16
  • Is it not possible to reverse the order of your decorators? – Nineteendo Dec 09 '22 at 09:21
  • 1
    Obviously the `second_func` calls the `first_func` :) That's what I refer to with going backwards. – apingaway Dec 09 '22 at 09:24
  • 1
    The calculus would then go as follows: in second_func: 2 * 10, then second_func's output is going to first_func as 20 * 20. How is that `second_func(first_func)` not backwards? – apingaway Dec 09 '22 at 09:26
  • "The second_func's output is going to first_func" — Well, in the sense that `first_func` calls `second_func` and receives its return value! – deceze Dec 09 '22 at 09:28
  • Or in Python [docs](https://wiki.python.org/moin/PythonDecorators) terms: > reverse order :) – apingaway Dec 09 '22 at 09:30
  • 1
    You need to make a more detailed case here, because it's a complex topic. There are *two* call chains going on: *when decorating*, and then again *when calling the decorated function.* Which of these do you think is "backwards" and why? How would you expect it to work differently? – deceze Dec 09 '22 at 09:33
  • The question boils down to why is the `num`'s output not passed to the `first_func`first but to the `second_func` instead? It refers to both cases, of which you ask me to single out one. I won't, because that is not the question. – apingaway Dec 09 '22 at 09:39
  • 1
    @apingaway The Python.org wiki is not Python docs. – AKX Dec 09 '22 at 09:39
  • 2
    `num`'s output isn't "passed" to anything. What happens when you call `num()` at the end is that you're calling `first_func`'s `inner`. That `inner` is now free to do whatever it wants. It decides to call `second_func`'s `inner`, which calls the function previously known as `num`. The result then gets returned back up the call chain. – deceze Dec 09 '22 at 09:42
  • 1
    How first_func calls second_func? Isn't the (func) refering to num() in this case? And what do you mean `num` doesn't output anything, as it clearly returns 10, which then gets multiplied by 2 in `second_func`? – apingaway Dec 09 '22 at 09:52
  • 1
    `first_func`'s `inner` calls `second_func`'s `inner` because it got passed `second_func`'s `inner` as parameter `func` at the time of decoration. — And I did not say `num` doesn't output anything; I said its output isn't "passed". That sounds like something calls `num` and takes its return value and calls `first_func` with it. Which is not what's happening. – deceze Dec 09 '22 at 09:54
  • So what's happening is, we can say for sure at least, that the calculation is starting in reversed order taking into account the way the decorators are listing the functions. Why is it starting to calculate from `second_function` and not from `first_function`? – apingaway Dec 09 '22 at 10:00
  • 1
    If by "calculation" you mean the final math operations, yes, they start "inside out" *because that's how you've written your decorators*. The decorators *first* call `x = func()` and *then* do some math with the result. That ends up as "inside out" operation. You *could* make your decorators do something *before* they call the decorated function. But of course, then they couldn't use the result of the decorated function in their calculation. – deceze Dec 09 '22 at 10:03

2 Answers2

5

Quoting the documentation on decorators:

The decorator syntax is merely syntactic sugar, the following two function definitions are semantically equivalent:

def f(arg):
   ...
f = staticmethod(f)
@staticmethod
def f(arg):
   ...

From this it follows that the decoration in

@a
@b
@c
def fun():
  ...

is equivalent to

fun = a(b(c(fun)))

IOW, it was designed like that because it's just syntactic sugar.

For proof, let's just decorate an existing function and not return a new one:

def dec1(f):
    print(f"dec1: got {vars(f)}")
    f.dec1 = True
    return f

def dec2(f):
    print(f"dec2: got {vars(f)}")
    f.dec2 = True
    return f

@dec1
@dec2
def foo():
    pass

print(f"Fully decked out: {vars(foo)}")

prints out

dec2: got {}
dec1: got {'dec2': True}
Fully decked out: {'dec2': True, 'dec1': True}
AKX
  • 152,115
  • 15
  • 115
  • 172
  • 1
    Not really correct answer, as the call order of the decorators is obviously c(b(a(fun))), as also noted [here](https://stackoverflow.com/questions/34173364/how-decorators-chaining-work). – apingaway Dec 09 '22 at 09:21
  • 2
    @apingaway "Call order" meaning what exactly? A decorator receives the decorated function as argument. So of course the innermost decoration must be evaluated first, otherwise the higher decorator couldn't receive it as argument. It's exactly the same as `a(b(c(func)))`. `c` is called first and passes its return value to `b`. `a` produces the final decorated function. When calling *that*, the call order is `a` → `b` → `c` → `fun`. – deceze Dec 09 '22 at 09:25
  • 4
    @apingaway I do think my answer is correct. You can also see an example of a similar chain [here in the Python docs](https://docs.python.org/3/reference/compound_stmts.html#function). – AKX Dec 09 '22 at 09:29
  • 1
    It is disputable on the basis of "reverse order"'s definition. [link](https://wiki.python.org/moin/PythonDecorators) – apingaway Dec 09 '22 at 09:32
  • 2
    AKX is clearly correct - inner functions must be actioned first. But @apingaway why not just watch it happening. Paste your code into https://pythontutor.com/visualize.html#mode=display and visualize execution to see the flow through the code. You can try other cases in the same way. – user19077881 Dec 09 '22 at 09:39
  • 1
    That's precisely what I did. – apingaway Dec 09 '22 at 09:43
  • 3
    Then you have seen the 'lowest' decorator returning before the higher just like an inner function before an outer function. The point of this argument alludes me - this is how it works because nothing else makes logical sense. The comment from @deceze well explains the process through the decorators. – user19077881 Dec 09 '22 at 09:54
2

TL;DR

g(f(x)) means applying f to x first, then applying g to the output.

Omit the parentheses, add @ before and line break after each function name:

@g
@f
x

(Syntax only valid if x is the definition of a function/class.)


Abstract explanation

The reasoning behind this design decision becomes fairly obvious IMHO, if you remember what the decorator syntax - in its most abstract and general form - actually means. So I am going to try the abstract approach to explain this.

It is all about syntax

To be clear here, the distinguishing factor in the concept of the "decorator" is not the object underneath it (so to speak) nor the operation it performs. It is the special syntax and the restrictions for it. Thus, a decorator at its core is nothing more than feature of Python grammar.

The decorator syntax requires a target to be decorated. Initially (see PEP 318) the target could only be function definitions; later class definitions were also allowed to be decorated (see PEP 3129).

Minimal valid syntax

Syntactically, this is valid Python:

def f(): pass

@f
class Target: pass  # or `def target: pass`

However, this will (perhaps unsuprisingly) cause a TypeError upon execution. As has been reiterated multiple times here and in other posts on this platform, the above is equivalent to this:

def f(): pass

class Target: pass

Target = f(Target)

Minimal working decorator

The TypeError stems from the fact that f lacks a positional argument. This is the obvious logical restriction imposed by what a decorator is supposed to do. Thus, to achieve not only syntactically valid code, but also have it run without errors, this is sufficient:

def f(x): pass

@f
class Target: pass

This is still not very useful, but it is enough for the most general form of a working decorator.

Decoration is just application of a function to the target and assigning the output to the target's name.

Chaining functions ⇒ Chaining decorators

We can ignore the target and what it is or does and focus only on the decorator. Since it merely stands for applying a function, the order of operations comes into play, as soon as we have more than one. What is the order of operation, when we chain functions?

def f(x): pass

def g(x): pass

class Target: pass

Target = g(f(Target))

Well, just like in the composition of purely mathematical functions, this implies that we apply f to Target first and then apply g to the result of f. Despite g appearing first (i.e. further left), it is not what is applied first.

Since stacking decorators is equivalent to nesting functions, it seems obvious to define the order of operation the same way. This time, we just skip the parentheses, add an @ symbol in front of the function name and a line break after it.

def f(x): pass

def g(x): pass

@g
@f
class Target: pass

But, why though?

If after the explanation above (and reading the PEPs for historic background), the reasoning behind the order of operation is still not clear or still unintuitive, there is not really any good answer left, other than "because the devs thought it made sense, so get used to it".


PS

I thought I'd add a few things for additional context based on all the comments around your question.

Decoration vs. calling a decorated function

A source of confusion seems to be the distinction between what happens when applying the decorator versus calling the decorated function.

Notice that in my examples above I never actually called target itself (the class or function being decorated). Decoration is itself a function call. Adding @f above the target is calling the f and passing the target to it as the first positional argument.

A "decorated function" might not even be a function

The distinction is very important because nowhere does it say that a decorator actually needs to return a callable (function or class). f being just a function means it can return whatever it wants. This is again valid and working Python code:

def f(x): return 3.14

@f
def target(): return "foo"

try:
    target()
except Exception as e:
    print(repr(e))
print(target)

Output:

TypeError("'float' object is not callable")
3.14

Notice that the name target does not even refer to a function anymore. It just holds the 3.14 returned by the decorator. Thus, we cannot even call target. The entire function behind it is essentially lost immediately before it is even available to the global namespace. That is because f just completely ignores its first positional argument x.

Replacing a function

Expanding this further, if we want, we can have f return a function. Not doing that seems very strange, considering it is used to decorate a function. But it doesn't have to be related to the target at all. Again, this is fine:

def bar(): return "bar"

def f(x): return bar

@f
def target(): return "foo"

print(target())
print(target is bar)

Output:

bar
True

It comes down to convention

The way decorators are actually overwhelmingly used out in the wild, is in a way that still keeps a reference to the target being decorated around somewhere. In practice it can be as simple as this:

def f(x):
    print(f"applied `f({x.__name__})`")
    return

@f
def target(): return "foo"

Just running this piece of code outputs applied f(target). Again, notice that we don't call target here, we only called f. But now, the decorated function is still target, so we could add the call print(target()) at the bottom and that would output foo after the other output produced by f.

The fact that most decorators don't just throw away their target comes down to convention. You (as a developer) would not expect your function/class to simply be thrown away completely, when you use a decorator.

Decoration with wrapping

This is why real-life decorators typically either return the reference to the target at the end outright (like in the last example) or they return a different callable, but that callable itself calls the target, meaning a reference to the target is kept in that new callable's local namespace . These functions are what is usually referred to as wrappers:

def f(x):
    print(f"applied `f({x.__name__})`")
    def wrapper():
        print(f"wrapper executing with {locals()=}")
        return x()
    return wrapper

@f
def target(): return "foo"

print(f"{target()=}")
print(f"{target.__name__=}")

Output:

applied `f(target)`
wrapper executing with locals()={'x': <function target at 0x7f1b2f78f250>}
target()='foo'
target.__name__='wrapper'

As you can see, what the decorator left us is wrapper, not what we originally defined as target. And the wrapper is what we call, when we write target().

Wrapping wrappers

This is the kind of behavior we typically expect, when we use decorators. And therefore it is not surprising that multiple decorators stacked together behave the way they do. The are called from the inside out (as explained above) and each adds its own wrapper around what it receives from the one applied before:

def f(x):
    print(f"applied `f({x.__name__})`")
    def wrapper_from_f():
        print(f"wrapper_from_f executing with {locals()=}")
        return x()
    return wrapper_from_f

def g(x):
    print(f"applied `g({x.__name__})`")
    def wrapper_from_g():
        print(f"wrapper_from_g executing with {locals()=}")
        return x()
    return wrapper_from_g

@g
@f
def target(): return "foo"

print(f"{target()=}")
print(f"{target.__name__=}")

Output:

applied `f(target)`
applied `g(wrapper_from_f)`
wrapper_from_g executing with locals()={'x': <function f.<locals>.wrapper_from_f at 0x7fbfc8d64f70>}
wrapper_from_f executing with locals()={'x': <function target at 0x7fbfc8d65630>}
target()='foo'
target.__name__='wrapper_from_g'

This shows very clearly the difference between the order in which the decorators are called and the order in which the wrapped/wrapping functions are called.

After the decoration is done, we are left with wrapper_from_g, which is referenced by our target name in global namespace. When we call it, wrapper_from_g executes and calls wrapper_from_f, which in turn calls the original target.

Daniil Fajnberg
  • 12,753
  • 2
  • 10
  • 41