Preserving signatures of decorated functions

Question

Suppose I have written a decorator that does something very generic. For example, it might convert all arguments to a specific type, perform logging, implement memoization, etc.

Here is an example:

def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    return g

@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z

>>> funny_function("3", 4.0, z="5")
22

Everything well so far. There is one problem, however. The decorated function does not retain the documentation of the original function:

>>> help(funny_function)
Help on function g in module __main__:

g(*args, **kwargs)

Fortunately, there is a workaround:

def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    g.__name__ = f.__name__
    g.__doc__ = f.__doc__
    return g

@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z

This time, the function name and documentation are correct:

>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(*args, **kwargs)
    Computes x*y + 2*z

But there is still a problem: the function signature is wrong. The information "*args, **kwargs" is next to useless.

What to do? I can think of two simple but flawed workarounds:

1 -- Include the correct signature in the docstring:

def funny_function(x, y, z=3):
    """funny_function(x, y, z=3) -- computes x*y + 2*z"""
    return x*y + 2*z

This is bad because of the duplication. The signature will still not be shown properly in automatically generated documentation. It's easy to update the function and forget about changing the docstring, or to make a typo. [And yes, I'm aware of the fact that the docstring already duplicates the function body. Please ignore this; funny_function is just a random example.]

2 -- Not use a decorator, or use a special-purpose decorator for every specific signature:

def funny_functions_decorator(f):
    def g(x, y, z=3):
        return f(int(x), int(y), z=int(z))
    g.__name__ = f.__name__
    g.__doc__ = f.__doc__
    return g

This works fine for a set of functions that have identical signature, but it's useless in general. As I said in the beginning, I want to be able to use decorators entirely generically.

I'm looking for a solution that is fully general, and automatic.

So the question is: is there a way to edit the decorated function signature after it has been created?

Otherwise, can I write a decorator that extracts the function signature and uses that information instead of "*kwargs, **kwargs" when constructing the decorated function? How do I extract that information? How should I construct the decorated function -- with exec?

Any other approaches?

Answer 1

1. Install decorator module:

   $ pip install decorator
   

2. Adapt definition of args_as_ints():

   import decorator
   
   @decorator.decorator
   def args_as_ints(f, *args, **kwargs):
       args = [int(x) for x in args]
       kwargs = dict((k, int(v)) for k, v in kwargs.items())
       return f(*args, **kwargs)
   
   @args_as_ints
   def funny_function(x, y, z=3):
       """Computes x*y + 2*z"""
       return x*y + 2*z
   
   print funny_function("3", 4.0, z="5")
   # 22
   help(funny_function)
   # Help on function funny_function in module __main__:
   # 
   # funny_function(x, y, z=3)
   #     Computes x*y + 2*z
   

---

Python 3.4+

functools.wraps() from stdlib preserves signatures since Python 3.4:

import functools


def args_as_ints(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return func(*args, **kwargs)
    return wrapper


@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z


print(funny_function("3", 4.0, z="5"))
# 22
help(funny_function)
# Help on function funny_function in module __main__:
#
# funny_function(x, y, z=3)
#     Computes x*y + 2*z

functools.wraps() is available at least since Python 2.5 but it does not preserve the signature there:

help(funny_function)
# Help on function funny_function in module __main__:
#
# funny_function(*args, **kwargs)
#    Computes x*y + 2*z

Notice: *args, **kwargs instead of x, y, z=3.

Answer 2

This is solved with Python's standard library functools and specifically functools.wraps function, which is designed to "update a wrapper function to look like the wrapped function". It's behaviour depends on Python version, however, as shown below. Applied to the example from the question, the code would look like:

from functools import wraps

def args_as_ints(f):
    @wraps(f) 
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    return g


@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z

When executed in Python 3, this would produce the following:

>>> funny_function("3", 4.0, z="5")
22
>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(x, y, z=3)
    Computes x*y + 2*z

Its only drawback is that in Python 2 however, it doesn't update function's argument list. When executed in Python 2, it will produce:

>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(*args, **kwargs)
    Computes x*y + 2*z

Answer 3

There is a decorator module with decorator decorator you can use:

@decorator
def args_as_ints(f, *args, **kwargs):
    args = [int(x) for x in args]
    kwargs = dict((k, int(v)) for k, v in kwargs.items())
    return f(*args, **kwargs)

Then the signature and help of the method is preserved:

>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(x, y, z=3)
    Computes x*y + 2*z

EDIT: J. F. Sebastian pointed out that I didn't modify args_as_ints function -- it is fixed now.

Answer 4

Take a look at the decorator module - specifically the decorator decorator, which solves this problem.

Answer 5

Second option:

1. Install wrapt module:

$ easy_install wrapt

wrapt have a bonus, preserve class signature.

import wrapt
import inspect

@wrapt.decorator
def args_as_ints(wrapped, instance, args, kwargs):
    if instance is None:
        if inspect.isclass(wrapped):
            # Decorator was applied to a class.
            return wrapped(*args, **kwargs)
        else:
            # Decorator was applied to a function or staticmethod.
            return wrapped(*args, **kwargs)
    else:
        if inspect.isclass(instance):
            # Decorator was applied to a classmethod.
            return wrapped(*args, **kwargs)
        else:
            # Decorator was applied to an instancemethod.
            return wrapped(*args, **kwargs)


@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x * y + 2 * z


>>> funny_function(3, 4, z=5))
# 22

>>> help(funny_function)
Help on function funny_function in module __main__:

funny_function(x, y, z=3)
    Computes x*y + 2*z

Answer 6

As commented above in jfs's answer ; if you're concerned with signature in terms of appearance (help, and inspect.signature), then using functools.wraps is perfectly fine.

If you're concerned with signature in terms of behavior (in particular TypeError in case of arguments mismatch), functools.wraps does not preserve it. You should rather use decorator for that, or my generalization of its core engine, named makefun.

from makefun import wraps

def args_as_ints(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        print("wrapper executes")
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return func(*args, **kwargs)
    return wrapper


@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z


print(funny_function("3", 4.0, z="5"))
# wrapper executes
# 22

help(funny_function)
# Help on function funny_function in module __main__:
#
# funny_function(x, y, z=3)
#     Computes x*y + 2*z

funny_function(0)  
# observe: no "wrapper executes" is printed! (with functools it would)
# TypeError: funny_function() takes at least 2 arguments (1 given)

See also this post about functools.wraps.

Answer 7

from inspect import signature


def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    sig = signature(f)
    g.__signature__ = sig
    g.__doc__ = f.__doc__
    g.__annotations__ = f.__annotations__
    g.__name__ = f.__name__
    return g

@args_as_ints
def funny_function(x, y, z=3):
    """Computes x*y + 2*z"""
    return x*y + 2*z

>>> funny_function("3", 4.0, z="5")
22

I wanted to add that answer (since this shows up first in google). The inspect module is able to fetch the signature of a function, so that it can be preserved in decorators. But that's not all. If you want to modify the signature, you can do so like this :

from inspect import signature, Parameter, _ParameterKind


def foo(a: int, b: int) -> int:
    return a + b

sig = signature(foo)
sig._parameters = dict(sig.parameters)
sig.parameters['c'] = Parameter(
    'c', _ParameterKind.POSITIONAL_OR_KEYWORD, 
    annotation=int
)
foo.__signature__ = sig

>>> help(foo)
Help on function foo in module __main__:

foo(a: int, b: int, c: int) -> int

Why would you want to mutate a function's signature ?

It's mostly useful to have adequate documentation on your functions and methods. If you're using the *args, **kwargs syntax and then popping arguments from kwargs for other uses in your decorators, that keyword argument won't be properly documented, hence, modifying the signature of the function.

Answer 8

def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    g.__name__ = f.__name__
    g.__doc__ = f.__doc__
    return g

this fixes name and documentation. to preserve the function signature, wrap is used exactly at same location as g.__name__ = f.__name__, g.__doc__ = f.__doc__.

the wraps itself a decorator. we pass the closure-the inner function to that decorator, and it is going to fix up the metadata. BUt if we only pass in the inner function to wraps, it is not gonna know where to copy the metadata from. It needs to know which function's metadata needs to be protected. It needs to know the original function.

def args_as_ints(f):
    def g(*args, **kwargs):
        args = [int(x) for x in args]
        kwargs = dict((k, int(v)) for k, v in kwargs.items())
        return f(*args, **kwargs)
    g=wraps(f)(g)
    return g

wraps(f) is going to return a function which will take g as its parameter. And that is going to return closure and will assigned to g and then we return it.