getting sphinx to recognise correct signature

Question

I've been trying to get my documentation in order for an open source project I'm working on, which involves a mirrored client and server API. To this end I have created a decorator that can most of the time be used to document a method that simply performs validation on its input. You can find a class full of these methods here and the decorator's implementation here.

The decorator, as you can see uses functools.wraps to preserve the docstring, and I thought also the signature, however the source code vs the generated documentation looks like this:

Source: source code

vs

Docs: sphinx docs

Does anyone know any way to have setH's generated documentation show the correct call signature? (without having a new decorator for each signature - there are hudreds of methods I need to mirror)

I've found a workaround which involved having the decorator not changing the unbound method, but having the class mutate the method at binding time (object instantiation) - this seems like a hack though, so any comments on this, or alternative ways of doing this, would be appreciated.

Well, technically, what is shown _is_ the new correct signature. You can pass any arguments to the decorated method. — Lev Levitsky, Jan 17 '13 at 16:46
Maybe there's something useful in this answer: http://stackoverflow.com/a/9349171/407651. — mzjn, Jan 17 '13 at 17:14
@mzjn almost but not quite - when I tried it I eventually found this at line 84 of decorator.py: `# func can be a class or a callable, but not an instance method` which kind of sucks, but I might have a look at implementing something similar for instance methods — theheadofabroom, Jan 17 '13 at 17:42
@LevLevitsky however if the original method does not accept the arguments it *will* throw a `TypeError` - it would be more accurate to say that the decorated method accepts a transformation of the original arguments - in this case a one-to-one mapping — theheadofabroom, Jan 17 '13 at 17:45
Note that with the advent of `__signature__`, this problem no longer occurs. — Eric, Jun 03 '20 at 13:16

score 5 · Answer 1 · answered Feb 25 '14 at 19:07

In PRAW, I handled this issue by having conditional decorators that return the original function (rather than the decorated function) when a sphinx build is occurring.

In PRAW's sphinx conf.py I added the following as a way to determine if SPHINX is currently building:

import os
os.environ['SPHINX_BUILD'] = '1'

And then in PRAW, its decorators look like:

import os

# Don't decorate functions when building the documentation
IS_SPHINX_BUILD = bool(os.getenv('SPHINX_BUILD'))

def limit_chars(function):
    """Truncate the string returned from a function and return the result."""
    @wraps(function)
    def wrapped(self, *args, **kwargs):
        output_string = function(self, *args, **kwargs)
        if len(output_string) > MAX_CHARS:
            output_string = output_string[:MAX_CHARS - 3] + '...'
        return output_string
    return function if IS_SPHINX_BUILD else wrapped

The return function if IS_SPHINX_BUILD else wrapped line is what allows SPHINX to pick up the correct signature.

Relevant Source

I love this solution (although it's a little bit of a hack), except for the fact that I am mostly using class decorators, so I still am unable to use this. For a temporary solution, I was going to have `__init__.py` do the wrapping only if `SPHINX_BUILD` is not set. This seems too convoluted, so I think I'll just deal with manually setting the signature in Sphinx, but it might be an idea someone else would be interested in. — Poik, Aug 20 '14 at 23:20
Like this solution. Only one comment: ``bool(os.getenv('SPHINX_BUILD'))`` will also be ``True`` for ``os.environ['SPHINX_BUILD'] = '0'``. Instead use something along the lines of ``IS_SPHINX_BUILD = os.getenv('SPHINX_BUILD') in ['1', 'True']`` — flinz, Mar 04 '16 at 14:13

theheadofabroom · Answer 2 · 2013-01-23T10:46:16.447

I'd like to avoid reliance on too muck outside of the standard library, so while I have looked at the Decorator module, I have mainly tried to reproduce its functionality.... Unsuccessfully...

So I took a look at the problem from another angle, and now I have a partially working solution, which can mainly be described by just looking at this commit. It's not perfect as it relies on using partial, which clobbers the help in the REPL. The idea is that instead of replacing the function to which the decorator is applied, it is augmented with attributes.

+def s_repr(obj):
+    """ :param obj: object """
+    return (repr(obj) if not isinstance(obj, SikuliClass)
+            else "self._get_jython_object(%r)" % obj._str_get)
+
+
 def run_on_remote(func):
     ...
-    func.s_repr = lambda obj: (repr(obj)
-                               if not isinstance(obj, SikuliClass) else
-                               "self._get_jython_object(%r)" % obj._str_get)
-
-    def _inner(self, *args):
-        return self.remote._eval("self._get_jython_object(%r).%s(%s)" % (
-            self._id,
-            func.__name__,
-            ', '.join([func.s_repr(x) for x in args])))
-
-    func.func = _inner
+    gjo = "self._get_jython_object"
+    func._augment = {
+        'inner': lambda self, *args: (self.remote._eval("%s(%r).%s(%s)"
+                                      % (gjo, self._id, func.__name__,
+                                         ', '.join([s_repr(x)for x in args]))))
+    }

     @wraps(func)
     def _outer(self, *args, **kwargs):
         func(self, *args, **kwargs)
-        if hasattr(func, "arg"):
-            args, kwargs = func.arg(*args, **kwargs), {}
-        result = func.func(*args, **kwargs)
-        if hasattr(func, "post"):
+        if "arg" in func._augment:
+            args, kwargs = func._augment["arg"](self, *args, **kwargs), {}
+        result = func._augment['inner'](self, *args, **kwargs)
+        if "post" in func._augment:
             return func.post(result)
         else:
             return result

     def _arg(arg_func):
-        func.arg = arg_func
-        return _outer
+        func._augment['arg'] = arg_func
+        return func

     def _post(post_func):
-        func.post = post_func
-        return _outer
+        func._augment['post'] = post_func
+        return func

     def _func(func_func):
-        func.func = func_func
-        return _outer
-    _outer.arg = _arg
-    _outer.post = _post
-    _outer.func = _func
-    return _outer
+        func._augment['inner'] = func_func
+        return func
+
+    func.arg  = _outer.arg = _arg
+    func.post = _outer.post = _post
+    func.func = _outer.func = _func
+    func.run  = _outer.run = _outer
+    return func

So this doesn't actually change the unbound method, ergo the generated documentation stays the same. The second part of the trickery occurs at class initialisation.

 class ClientSikuliClass(ServerSikuliClass):
     """ Base class for types based on the Sikuli native types """
     ...
     def __init__(self, remote, server_id, *args, **kwargs):
         """
         :type server_id: int
         :type remote: SikuliClient
         """
         super(ClientSikuliClass, self).__init__(None)
+        for key in dir(self):
+            try:
+                func = getattr(self, key)
+            except AttributeError:
+                pass
+            else:
+                try:
+                    from functools import partial, wraps
+                    run = wraps(func.run)(partial(func.run, self))
+                    setattr(self, key, run)
+                except AttributeError:
+                    pass
         self.remote = remote
         self.server_id = server_id

So at the point where an instance of any class inheriting ClientSikuliClass is instantiated, an attempt is made to take the run property of each attribute of that instance and make that what is returned on attempting to get that attribute, and so the bound method is now a partially applied _outer function.

So the issues with this are multiple:

Using partial at initilaisation results in losing the bound method information.
I worry about clobbering attributes that just so happen to have a run attribute...

So while I have an answer to my own question, I'm not quite satisfied by it.

Update

Ok so after a bit more work I ended up with this:

 class ClientSikuliClass(ServerSikuliClass):
     """ Base class for types based on the Sikuli native types """
     ...
     def __init__(self, remote, server_id, *args, **kwargs):
         """
         :type server_id: int
         :type remote: SikuliClient
         """
         super(ClientSikuliClass, self).__init__(None)
-        for key in dir(self):
+
+        def _apply_key(key):
             try:
                 func = getattr(self, key)
+                aug = func._augment
+                runner = func.run
             except AttributeError:
-                pass
-            else:
-                try:
-                    from functools import partial, wraps
-                    run = wraps(func.run)(partial(func.run, self))
-                    setattr(self, key, run)
-                except AttributeError:
-                    pass
+                return
+
+            @wraps(func)
+            def _outer(*args, **kwargs):
+                return runner(self, *args, **kwargs)
+
+            setattr(self, key, _outer)
+
+        for key in dir(self):
+            _apply_key(key)
+
         self.remote = remote
         self.server_id = server_id

This prevents the loss of the documentation on the object. You'll also see that the func._augment attribute is accessed, even though it is not used, so that if it does not exist the object attribute will not be touched.

I'd be interested if anyone had any comments on this?

score 3 · Answer 3 · answered Jan 21 '13 at 22:34

3

functools.wraps only preserves __name__,__doc__, and __module__. To preserve the signature as well take a look at Michele Simionato's Decorator module.

answered Jan 21 '13 at 22:34

Ethan Furman

63,992
20
159
237

Not quite right for my use case, but might be useful to others – theheadofabroom Jan 29 '13 at 09:52
1

I believe the Decorator module is for use with cPython; not sure what magic to use for Jython. – Ethan Furman Jan 29 '13 at 19:02
Ethan's suggestion worked for me in the same situation as described by the OP. – Andreas Maier Aug 29 '16 at 15:05

score 2 · Accepted Answer · edited May 23 '17 at 10:33

To expand my short comment on Ethan's answer, here is my original code using the functools package:


import functools

def trace(f):
    """The trace decorator."""
    logger = ... # some code to determine the right logger
    where = ... # some code to create a string describing where we are

    @functools.wraps(f)
    def _trace(*args, **kwargs):
        logger.debug("Entering %s", where)
        result = f(*args, **kwargs)
        logger.debug("Leaving %s", where)
        return result

    return _trace

and here the code using the decorator package:


import decorator

def trace(f):
    """The trace decorator."""
    logger = ... # some code to determine the right logger
    where = ... # some code to create a string describing where we are

    def _trace(f, *args, **kwargs):
        logger.debug("Entering %s", where)
        result = f(*args, **kwargs)
        logger.debug("Leaving %s", where)
        return result

    return decorator.decorate(f, _trace)

We wanted to move the code to determine the right logger and where-string out of the actual function wrapper, for performance reasons. Hence the approach with the nested wrapper function, in both versions.

Both versions of the code work on Python 2 and Python 3, but the second version creates the correct prototypes for the decorated functions when using Sphinx & autodoc (without having to repeat the prototype in the autodoc statements, as suggested in this answer).

This is with cPython, I did not try Jython etc.

getting sphinx to recognise correct signature

4 Answers4

Update

Linked