decorating decorators: try to get my head around understanding it

Question

I'm trying to understand how to decorate decorators, and wanted to try out the following:

Let's say I have two decorators and apply them to the function hello():

def wrap(f):
    def wrapper():
        return " ".join(f())
    return wrapper


def upper(f):
    def uppercase(*args, **kargs):
        a,b = f(*args, **kargs)
        return a.upper(), b.upper()
    return uppercase

@wrap
@upper
def hello():
    return "hello","world"

print(hello())

Then I have to start adding other decorators for other functions, but in general the @wrap decorator will "wrap all of them"

def lower(f):
    def lowercase(*args, **kargs):
        a,b = f(*args, **kargs)
        return a.lower(), b.lower()
    return lowercase

@wrap
@lower
def byebye():
    return "bye", "bye"

How do I write a decorator, which decorates my @lower and @upper decorators? See below:

@wrap
def lower():
    ...

@wrap
def upper():
    ...

To achieve the same result as above by only doing:

@upper
def hello():
    ...

@lower
def byebye():
    ...

Answer 1

def upper(f):
    @wrap
    def uppercase(*args, **kargs):
        a,b = f(*args, **kargs)
        return a.upper(), b.upper()
    return uppercase

---

A decorator in Python

 @foo
 def bar(...): ...

is just equivalent to

 def bar(...): ...
 bar = foo(bar)

You want to get the effect of

@wrap
@upper
def hello(): ....

i.e.

hello = wrap(upper(hello))

so the wrap should be called on the return value of upper:

def upper_with_wrap(f):
   def uppercase(...): ...
   return wrap(uppercase)

which is also equivalent to applying the decorator on that function:

def upper_with_wrap(f):
   @wrap
   def uppercase(...): ...
   # ^ equivalent to 'uppercase = wrap(uppercase)'
   return uppercase

Answer 2

Here's a generic (and slightly convoluted) solution for decorating decorators with decorators (Yay!).

# A second-order decorator
def decdec(inner_dec):
    def ddmain(outer_dec):
        def decwrapper(f):
            wrapped = inner_dec(outer_dec(f))
            def fwrapper(*args, **kwargs):
               return wrapped(*args, **kwargs)
            return fwrapper
        return decwrapper
    return ddmain

def wrap(f):
    def wrapper():
        return " ".join(f())
    return wrapper


# Decorate upper (a decorator) with wrap (another decorator)
@decdec(wrap)
def upper(f):
    def uppercase(*args, **kargs):
        a,b = f(*args, **kargs)
        return a.upper(), b.upper()
    return uppercase

@upper
def hello():
    return "hello","world"

print(hello())

Answer 3

The accepted answer from Boaz Yaniv can be simplified a little bit like this:

def wrap(decorator):                # lowercase/uppercase decorator as argument
    def wrapperA(fn):               # decorated (hello) function as argument
        def wrapperB(*fn_args):     # decorated (hello) functions' arguments as arguments
            # Here, you tell your 'wrap' decorator to get the
            # result that we want to process/wrap further this way:
            t = decorator(fn)(*fn_args)
            return " ".join(t)
        return wrapperB
    return wrapperA


@wrap
def lower(f):
    def lowercase(*args, **kwargs):
        a, b = f(*args, **kwargs)
        return a.lower(), b.lower(), 'in lowercase'
    return lowercase


@wrap
def upper(f):
    def uppercase(*args, **kwargs):
        a, b = f(*args, **kwargs)
        return a.upper(), b.upper(), 'in uppercase'
    return uppercase


@lower
def hello_in_lc():
    return "HELLO", "WORLD"


@upper
def hello_in_uc():
    return "hello", "world"


x = hello_in_lc()
y = hello_in_uc()
print(x)
print(y)

Output:

hello world in lowercase
HELLO WORLD in uppercase

Now, the reason the wrap decorator has so much nesting is because once you decorate another decorator's definition, the whole execution process becomes a bit different than the vertical nested decoration. Here's what I am talking about:

@wrap        # this wrap decorator is the one provided in the question
@upper
def hello()
    return "Hello", "World"

Here's what Python does with the above code. It calls the wrap decorator with upper function as argument. Inside wrapper of wrap decorator it finds that the decorator is being invoked. So it invokes it, the upper function. Because upper itself is a decorator of a function, during its invocation upper receives hello function as a reference and its arguments are received by wrapper uppercase.

Inside uppercase a call to hello function is made, so hello function is then invoked, its result is processed, and is returned back to wrapper function in wrap decorator which is finally returned back to main module's global scope where hello() was invoked at the bottom.

However, decorating a decorator's definition is a different story. Take the following code:

@wrap        # that's the wrap decorator from my answer
def upper()

@upper
def hello()

What's gonna happen here is once you invoke hello() the wrap decorator would be invoked like in the earlier situation, with upper function passed as an argument to wrap. However, if you try to invoke decorator inside wrapper as decorator() Python would throw error that upper function was called without an argument!

To fix that your wrapper (called wrapperA here) in wrap decorator needs to receive an argument. This argument is the reference to the function that upper has decorated, which is hello function in our case. So the wrapperA has to call the decorator with hello function (fn) as argument.

But executing decorator(fn) would give us the reference to upper decorator's uppercase wrapper. In order to execute it and to pass any argument that fn would need we need another wrapper called wrapperB nested in wrapperA. Here, wrapper A would literally act as a decorator of fn and its wrapperB would take fn's arguments.

So the compounded final call to get the result for any processing on intermediate decorators' (lower and upper) result should look like this:

def wrapperB(*fn_args):
    decorator(fn)(*fn_args)

# Alternative
    wrapped_fn = decorator(fn)
    result = wrapped_fn(*fn_args) 

And that's why so must nesting is needed.

Vertical decorating or the alternative suggested by kennytm here is obviously much better on the eyes.