How do I use line_profiler (from Robert Kern)?

Question

I have tried using the line_profiler module for getting a line-by-line profile over a Python file. This is what I've done so far:

1) Installed line_profiler from pypi by using the .exe file (I am on WinXP and Win7). Just clicked through the installation wizard.

2) Written a small piece of code (similar to what has been asked in another answered question here).

from line_profiler import LineProfiler
def do_stuff(numbers):
    print numbers

numbers = 2
profile = LineProfiler(do_stuff(numbers))
profile.print_stats()

3) Run the code from IDLE/PyScripter. I got only the time.

Timer unit: 4.17188e-10 s

How do I get full line-by-line profile over the code I execute? I have never used any advanced Python features like decorators, so it is hard for me to understand how shall I use the guidelines provided by several posts like here and here.

score 84 · Answer 1 · edited May 23 '17 at 12:10

This answer is a copy of my answer here for how to get line_profiler statistics from within a Python script (without using kernprof from the command line or having to add @profile decorators to functions and class methods). All answers (that I've seen) to similar line_profiler questions only describe using kernprof.

The line_profiler test cases (found on GitHub) have an example of how to generate profile data from within a Python script. You have to wrap the function that you want to profile and then call the wrapper passing any desired function arguments.

from line_profiler import LineProfiler
import random

def do_stuff(numbers):
    s = sum(numbers)
    l = [numbers[i]/43 for i in range(len(numbers))]
    m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()

Output:

Timer unit: 1e-06 s

Total time: 0.000649 s
File: <ipython-input-2-2e060b054fea>
Function: do_stuff at line 4

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     4                                           def do_stuff(numbers):
     5         1           10     10.0      1.5      s = sum(numbers)
     6         1          186    186.0     28.7      l = [numbers[i]/43 for i in range(len(numbers))]
     7         1          453    453.0     69.8      m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

Adding Additional Functions to Profile

Also, you can add additional functions to be profiled as well. For example, if you had a second called function and you only wrap the calling function, you'll only see the profile results from the calling function.

from line_profiler import LineProfiler
import random

def do_other_stuff(numbers):
    s = sum(numbers)

def do_stuff(numbers):
    do_other_stuff(numbers)
    l = [numbers[i]/43 for i in range(len(numbers))]
    m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()

The above would only produce the following profile output for the calling function:

Timer unit: 1e-06 s

Total time: 0.000773 s
File: <ipython-input-3-ec0394d0a501>
Function: do_stuff at line 7

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     7                                           def do_stuff(numbers):
     8         1           11     11.0      1.4      do_other_stuff(numbers)
     9         1          236    236.0     30.5      l = [numbers[i]/43 for i in range(len(numbers))]
    10         1          526    526.0     68.0      m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

In this case, you can add the additional called function to profile like this:

from line_profiler import LineProfiler
import random

def do_other_stuff(numbers):
    s = sum(numbers)

def do_stuff(numbers):
    do_other_stuff(numbers)
    l = [numbers[i]/43 for i in range(len(numbers))]
    m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

numbers = [random.randint(1,100) for i in range(1000)]
lp = LineProfiler()
lp.add_function(do_other_stuff)   # add additional function to profile
lp_wrapper = lp(do_stuff)
lp_wrapper(numbers)
lp.print_stats()

Output:

Timer unit: 1e-06 s

Total time: 9e-06 s
File: <ipython-input-4-dae73707787c>
Function: do_other_stuff at line 4

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     4                                           def do_other_stuff(numbers):
     5         1            9      9.0    100.0      s = sum(numbers)

Total time: 0.000694 s
File: <ipython-input-4-dae73707787c>
Function: do_stuff at line 7

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     7                                           def do_stuff(numbers):
     8         1           12     12.0      1.7      do_other_stuff(numbers)
     9         1          208    208.0     30.0      l = [numbers[i]/43 for i in range(len(numbers))]
    10         1          474    474.0     68.3      m = ['hello'+str(numbers[i]) for i in range(len(numbers))]

NOTE: Adding functions to profile in this way does not require changes to the profiled code (i.e., no need to add @profile decorators).

Good answer! But I actually find the decorators a good way to control profiling. Is there a way to use line_prof with decorators within-script (without kernprof)? — A. Donda, May 14 '19 at 16:51
I'm not 100% this works. I haven't tested it yet. Thanks to tdube for the line_profiler code. All I did was convert an existing decorator I had to hopefully work with LineProfiler. EDIT: formatting was messed up because I can't figure out how to format code in comments. [Here's a Pastebin.](https://pastebin.com/vgFhxjxF) — brettroyerr, Apr 17 '20 at 08:46
That's what I'm using currently in my Django code (where I can't use command line `kernprof`): https://stackoverflow.com/a/68163807/1937033 — ThePhi, Jun 29 '21 at 04:38

martineau · Accepted Answer · 2018-11-27T11:45:20.643

20

Just follow Dan Riti's example from the first link, but use your code. All you have to do after installing the line_profiler module is add a @profile decorator right before each function you wish to profile line-by-line and make sure each one is called at least once somewhere else in the code—so for your trivial example code that would be something like this:

example.py file:

@profile
def do_stuff(numbers):
    print numbers

numbers = 2
do_stuff(numbers)

Having done that, run your script via the kernprof.py^✶ that was installed in your C:\Python27\Scripts directory. Here's the (not very interesting) actual output from doing this in a Windows 7 command-line session:

> python "C:\Python27\Scripts\kernprof.py" -l -v example.py
2
Wrote profile results to example.py.lprof
Timer unit: 3.2079e-07 s

File: example.py
Function: do_stuff at line 2
Total time: 0.00185256 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           @profile
     2                                           def do_stuff(numbers):
     3         1         5775   5775.0    100.0      print numbers

You likely need to adapt this last step—the running of your test script with kernprof.py instead of directly by the Python interpreter—in order to do the equivalent from within IDLE or PyScripter.

^✶Update

It appears that in line_profiler v1.0, the kernprof utility is distributed as an executable, not a .py script file as it was when I wrote the above. This means the following now needs to used to invoke it from the command-line:

> "C:\Python27\Scripts\kernprof.exe" -l -v example.py

edited Nov 27 '18 at 11:45

answered May 27 '14 at 12:01

martineau

119,623
25
170
301

4

You're welcome. BTW, if you want to run the script normally (without `kernprof.py`) you'll need to remove the `@profile` decorator call(s) or define your own dummy one: e.g. `profile = lambda f: f` at the beginning of the file. – martineau May 27 '14 at 12:32
This isn't working for me because I don't seem to be able to call kernprof.py. I used pip install to get line_profiler and a file called kernprof is there but it doesn't have the .py extension, do you know why this would be? Thank you. – Alex Sep 25 '15 at 22:00
@Alex: No idea...could be a bad install. Try it again and it that doesn't work do it manually without `pip` by downloading the module's source from [pypi](https://pypi.python.org/pypi/line_profiler/1.0). – martineau Sep 26 '15 at 03:14
5

@Alex `kernprof` doesn't have to have `.py` extension to be executable. Mine installs inside a venv and runs fine as simply `kernprof`. – rszalski Jul 19 '16 at 12:55
@martineau Do you know why the original code didn't gather statistics? It is a minimal example and the invocation "looks ok" to me. Seems to run fine when one replaces `LineProfiler(..)` with `cProfile.Profile(...)`. – rszalski Jul 19 '16 at 13:05
@szalski: I assume it's because the `@profile` decorator wasn't applied to any functions, so no statistics were gathered to print. You might be able to figure it out by looking at the source code. – martineau Jul 19 '16 at 15:19
@everyone: There's a newer version of Kern's `line_profiler` utility available (currently 2.1.2). You can get it from [pypi](https://pypi.org/search/?q=line_profiler) or from the project's [homepage](https://github.com/rkern/line_profiler). From what's shown in the change-log there, it looks like it's supported Python 3 since the first non-beta release of version 1.0. – martineau Nov 27 '18 at 12:17
For Python 3.10, use this command: `python -m kernprof -l -v "script with things added for profiler.py"`and it prints the result to the command line as expected. (I installed kernprof today using `pip install line_profiler`) – Petr L. Jul 19 '23 at 10:18

score 8 · Answer 3 · answered Nov 01 '21 at 02:50

Found a good use to line_profiler using decorator i.e. @profile that worked for me:

def profile(func):
    from functools import wraps

    @wraps(func)
    def wrapper(*args, **kwargs):
        from line_profiler import LineProfiler
        prof = LineProfiler()
        try:
            return prof(func)(*args, **kwargs)
        finally:
            prof.print_stats()

    return wrapper

Credits to: pavelpatrin

Mahmoud Al-Bassam · Answer 4 · 2021-05-08T12:47:21.610

6

load the line_profiler and numpy

%load_ext line_profiler
import numpy as np

define a function for example:

def take_sqr(array):
    sqr_ar = [np.sqrt(x) for x in array]
    return sqr_ar

use line_profiler to count the time as follows:

%lprun -f take_sqr take_sqr([1,2,3])

the output looks like this:

Timer unit: 1e-06 s
 
Total time: 6e-05 s File: <ipython-input-5-e50c1b05a473> Function:
take_sqr at line 1

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     1                                           def take_sqr(array):
     2         4         59.0     14.8     98.3      sqr_ar = [np.sqrt(x) for x in array]
     3         1          1.0      1.0      1.7      return sqr_ar

edited May 08 '21 at 12:47

answered May 07 '21 at 05:44

Mahmoud Al-Bassam

61
1
2

3

Welcome to Stack Overflow! Code-only answers are not particularly helpful. Please include a brief description of how this code solves the problem. – 4b0 May 07 '21 at 06:06
Thank you! I just edited the response. I hope you find it clearer now. – Mahmoud Al-Bassam May 08 '21 at 12:48
I used just this way but I would like to suppress that output because I don't need that by now (only need output during debug) – JOSMAR BARBOSA - M4NOV3Y May 24 '23 at 00:13

score 4 · Answer 5 · answered Apr 24 '21 at 09:23

4

If you're using PyCharm, you can also take a look at https://plugins.jetbrains.com/plugin/16536-line-profiler

It's a plugin I created that allows you to load and visualize line profiler results into the PyCharm editor.

answered Apr 24 '21 at 09:23

jusx

1,099
9
14

Do you have any manual on how to remove the values once you want to retake coding?, congrats by the way, this is pretty cool. – MBV Sep 11 '22 at 19:11

score 2 · Answer 6 · answered Nov 06 '22 at 18:40

Just an addition to @Lhenkel answer. This is a decorator for async functions

def async_profile(func):
    """line profiler for an async funciton"""
    from functools import wraps

    @wraps(func)
    async def wrapper(*args, **kwargs):
        from line_profiler import LineProfiler
        prof = LineProfiler()
        try:
            return await prof(func)(*args, **kwargs)
        finally:
            prof.print_stats()

    return wrapper

To use these decorators with methods read this answer

How do I use line_profiler (from Robert Kern)?

6 Answers6

load the line_profiler and numpy

define a function for example:

use line_profiler to count the time as follows:

the output looks like this:

Linked