15

I want to use the excellent line_profiler, but only some of the time. To make it work I add

@profile

before every function call, e.g.

@profile
def myFunc(args):
    blah
    return

and execute

kernprof.py -l -v mycode.py args

But I don't want to have to put the @profile decorators in by hand each time, because most of the time I want to execute the code without them, and I get an exception if I try to include them, e.g.

mycode.py args

Is there a happy medium where I can dynamically have the decorators removed based on some condition switch/argument, without having to do things manually and/or modify each function too much?

Peter Mortensen
  • 30,738
  • 21
  • 105
  • 131
jtlz2
  • 7,700
  • 9
  • 64
  • 114
  • I'd ask myself if I really need to profile so frequently that it needs such support. I'm not saying you do or don't, so you needn't answer. I just found the use case a little surprising. – msw Aug 14 '13 at 11:01
  • The code takes a long time (hours at the moment...) to execute, so for now I want to kill two birds with one stone in getting results and profiling simultaneously. I suppose I see profiling as an ongoing process (since I'm new to/excited about it), so I wouldn't just use it in the (many) functions, declare it done and remove all the decorators. – jtlz2 Aug 14 '13 at 21:20
  • I wouldn't let something take hours without [*trying this*](http://stackoverflow.com/a/4299378/23771). It costs nothing and tells you exactly what's going on. – Mike Dunlavey Aug 29 '15 at 16:10
  • 1
    @jtlz2 You can also wrap functions and class methods to profile (possibly in a separate profiling script) and avoid adding `@profile` decorators entirely as described here (https://stackoverflow.com/a/43376466/5874320). – tdube Sep 02 '17 at 21:01

5 Answers5

23

Instead of removing the @profile decorator lines, provide your own pass-through no-op version.

You can add the following code to your project somewhere:

try:
    # Python 2
    import __builtin__ as builtins
except ImportError:
    # Python 3
    import builtins

try:
    builtins.profile
except AttributeError:
    # No line profiler, provide a pass-through version
    def profile(func): return func
    builtins.profile = profile

Import this before any code using the @profile decorator and you can use the code with or without the line profiler being active.

Because the dummy decorator is a pass-through function, execution performance is not impacted (only import performance is every so lightly affected).

If you don't like messing with built-ins, you can make this a separate module; say profile_support.py:

try:
    # Python 2
    import __builtin__ as builtins
except ImportError:
    # Python 3
    import builtins

try:
    profile = builtins.profile
except AttributeError:
    # No line profiler, provide a pass-through version
    def profile(func): return func

(no assignment to builtins.profile) and use from profile_support import profile in any module that uses the @profile decorator.

Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • The issue with this is that if you want to "prime" your run with something like `kernprof -l --setup script.py script.py`, It will overwrite the @profile decorator with noop for both runs. – JonnyRobbie Oct 04 '21 at 07:58
  • @JonnyRobbie: then use another decorator that determines if `builtins.profile` exists when the script is run, not when it is imported. What happens with `--setup` is that the code is imported with `builtins.profile` not set, all your code is decorated with the no-op decorator, and so nothing is being profiled on the actual profiling run. – Martijn Pieters Oct 29 '21 at 12:52
9

You don't need to import __builtins__/builtins or LineProfiler at all, you can simply rely on a NameError when trying to lookup profile:

try:
    profile
except NameError:
    profile = lambda x: x

However this needs to be included in every file that uses profile, but it doesn't (permanently) alter the global state (builtins) of Python.

MSeifert
  • 145,886
  • 38
  • 333
  • 352
6

A comment that grew to become a variant of @Martijin Pieters answer.

I prefer not to involve __builtin__ at all. W/o a comment, it would be practically impossible for someone else to guess that line_profiler is involved, w/o a priori knowing this.

Looking at kernprof line 199, it suffices to instantiate LineProfiler.

try:
    from line_profiler import LineProfiler
    profile = LineProfiler()
except ImportError:
    def profile(func):
        return func

Importing (explicit) is better than globally modifying builtins (implicit). If the profiling decorators are permanent, then their origin should be clear in the code itself.

In presence of line_profiler, the above approach will wrap the decorated functions with profilers on every run, irrespective of whether run by kernprof. This side-effect may be undesired.

0 _
  • 10,524
  • 11
  • 77
  • 109
1

I am using the following modified version with Python 3.4

try:
    import builtins
    profile = builtins.__dict__['profile']
except KeyError:
    # No line profiler, provide a pass-through version
    def profile(func): return func
ChrisP
  • 5,812
  • 1
  • 33
  • 36
0

The other answers are correct, but those may pose an issue when trying to "prime" the script with something like kernprof -l --setup script.py script.py. Priming your script like this may be useful for example when you try to optimize your functions with numba and you don't want to bias your line timings with compiling (or loading from filesystem cache, which still has a significant overhead even with numbas cache=True param).

The issue is that the setup run noops all your @profile decorators and renders them moot for the profiling run.

I solved that by moving the try except to the actual decorator run like this:

def profile2(f):
    def s(*args, **kwargs):
        try:
            return profile(f)(*args, **kwargs)
        except NameError:
            return f(*args, **kwargs)
    return s

and decorating all my profiled function with @profile2.

JonnyRobbie
  • 526
  • 5
  • 16