12

I'm using the retry decorator in some code in python. But I want to speed up my tests by removing its effect.

My code is:

@retry(subprocess.CalledProcessError, tries=5, delay=1, backoff=2, logger=logger)
def _sftp_command_with_retries(command, pem_path, user_at_host):
    # connect to sftp, blah blah blah
    pass

How can I remove the effect of the decorator while testing? I can't create an undecorated version because I'm testing higher-level functions that use this.

Since retry uses time.sleep to back off, ideally I'd be able to patch time.sleep but since this is in a decorator I don't think that's possible.

Is there any way I can speed up testing code that uses this function?

Update

I'm basically trying to test my higher-level functions that use this to make sure that they catch any exceptions thrown by _sftp_command_with_retries. Since the retry decorator will propagate them I need a more complicated mock.

So from here I can see how to mock a decorator. But now I need to know how to write a mock that is itself a decorator. It needs to call _sftp_command_with_retries and if it raises an exception, propagate it, otherwise return the return value.

Adding this after importing my function didn't work:

_sftp_command_with_retries = _sftp_command_with_retries.__wrapped__
Community
  • 1
  • 1
jbrown
  • 7,518
  • 16
  • 69
  • 117
  • 1
    Why not patch out `retry` entirely? It could be as simple as: `lambda *args, **kwargs: lambda func: func`, then you can just test the decorated function directly. – jonrsharpe Sep 21 '15 at 14:32
  • 1
    How can you patch it before it's applied? That's the problem with decorators - when you import the file that includes a decorated function it's applied to the function immediately AFAIK. So you don't actually have a chance to mock it before importing it like you do with other functions. If I'm wrong feel free to write an answer showing how to mock it out. I'm not trying to test `retry`'s behaviour here. – jbrown Sep 21 '15 at 14:36
  • If that's the part of the problem you're stuck on, that duplicate should cover you. If not, please update the question to clarify. – jonrsharpe Sep 21 '15 at 14:37
  • The `retry` decorator uses the `decorator` package if installed. This means you can just use `_sftp_command_with_retries.__wrapped__` in that case to reach the original. – Martijn Pieters Sep 21 '15 at 14:41
  • @jonrsharpe so it turns out this isn't an exact dupe because I need the behaviour that retry provides in that it will pass up any exceptions that are caught in it. That's what I'm testing. But it does help. I now need to write a mock that will propagate exceptions... – jbrown Sep 21 '15 at 14:47
  • @jbrown then could you please edit the question accordingly – jonrsharpe Sep 21 '15 at 14:47
  • @MartijnPieters I tried that but it didn't make any difference. After importing my function at the top of my file I added `_sftp_command_with_retries = _sftp_command_with_retries.__wrapped__` but they aren't any faster. I installed the `decorator` package as well. – jbrown Sep 21 '15 at 14:51
  • @jbrown: And did you actually call the unwrapped version in your test or did you use an indirect reference somewhere? – Martijn Pieters Sep 21 '15 at 14:55
  • Just for the record... I've already answer to this question at http://stackoverflow.com/a/30016312/4101725 – Michele d'Amico Sep 21 '15 at 21:03

1 Answers1

12

The retry decorator you are using is built on top of the decorator.decorator utility decorator with a simpler fallback if that package is not installed.

The result has a __wrapped__ attribute that gives you access to the original function:

orig = _sftp_command_with_retries.__wrapped__

If decorator is not installed and you are using a Python version before 3.2, that attribute won't be present; you'd have to manually reach into the decorator closure:

orig = _sftp_command_with_retries.__closure__[1].cell_contents

(the closure at index 0 is the retry_decorator produced when calling retry() itself).

Note that decorator is listed as a dependency in the retry package metadata, and if you installed it with pip the decorator package would have been installed automatically.

You can support both possibilities with a try...except:

try:
    orig = _sftp_command_with_retries.__wrapped__
except AttributeError:
    # decorator.decorator not available and not Python 3.2 or newer.
    orig = _sftp_command_with_retries.__closure__[1].cell_contents

Note that you always can patch time.sleep() with a mock. The decorator code will use the mock as it references the 'global' time module in the module source code.

Alternatively, you could patch retry.api.__retry_internal with:

import retry.api
def dontretry(f, *args, **kw):
    return f()

with mock.patch.object(retry.api, '__retry_internal', dontretry):
    # use your decorated method

This temporarily replaces the function that does the actual retrying with one that just calls your original function directly.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Great answer. Thanks. Patching `retry.api.__retry_internal` worked. – jbrown Sep 21 '15 at 15:05
  • Great answer... but this question is a clear duplicate of http://stackoverflow.com/questions/29996592/how-to-mock-function-call-used-by-imported-pypi-library-in-python/30016312#30016312 . Our answer are little bit different (your is better but my discourage to use some kind of hacking and encourage to change design). What we should do? mark it as duplicate or leave all as it is? – Michele d'Amico Sep 21 '15 at 21:10
  • @Micheled'Amico: indeed, it is a duplicate. I've closed this one now as such. If you see a duplicate like this always feel free to vote to close it as such! :-) – Martijn Pieters Sep 22 '15 at 07:16