1

I want to profile the time used for every line in a specific function export(), which is in the IO part of my library and thus has the path mylib.io.export(). In order to test it with real world data I need to run load_transform_export.py, which more or less contains the following code:

import mylib

i_data = mylib.io.read(...)
t_data = mylib.utils.transform(i_data)
train, test = mylib.utils.split(t_data)
mylib.io.export(train, 'train.h5')
mylib.io.export(test, 'test.h5')

I've tried using line_profiler by adding @profile to mylib.io.export() and running the script with kernprof -l -v load_transform_export.py. However, it seems that it cannot inject into a function in a library (e.g. not the main script). How can I profile the time used for each line in my function with this setup?

Edit:

The output for the profiling should be per-line basis, and therefore similar to this:

Wrote profile results to primes.py.lprof
Timer unit: 1e-06 s

File: primes.py
Function: primes at line 2
Total time: 0.00019 s

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     2                                           @profile
     3                                           def primes(n): 
     4         1            2      2.0      1.1      if n==2:
     5                                                   return [2]
     6         1            1      1.0      0.5      elif n<2:
     7                                                   return []
     8         1            4      4.0      2.1      s=range(3,n+1,2)
     9         1           10     10.0      5.3      mroot = n ** 0.5
    10         1            2      2.0      1.1      half=(n+1)/2-1
    11         1            1      1.0      0.5      i=0
    12         1            1      1.0      0.5      m=3
    13         5            7      1.4      3.7      while m <= mroot:
    14         4            4      1.0      2.1          if s[i]:
    15         3            4      1.3      2.1              j=(m*m-3)/2
    16         3            4      1.3      2.1              s[j]=0
    17        31           31      1.0     16.3              while j<half:
    18        28           28      1.0     14.7                  s[j]=0
    19        28           29      1.0     15.3                  j+=m
    20         4            4      1.0      2.1          i=i+1
    21         4            4      1.0      2.1          m=2*i+3
    22        50           54      1.1     28.4      return [2]+[x for x in s if x]
pir
  • 5,513
  • 12
  • 63
  • 101
  • Can you post a bit of code to better explain what you are doing? – Bretsky Sep 04 '15 at 09:00
  • Sure. However, I don't quite know what exactly needs more explanation :) – pir Sep 04 '15 at 09:15
  • Are you putting `@profile` on the `def export()` in `mylib.io`, or when you are calling it in `load_transform_export.py`? – Bretsky Sep 04 '15 at 09:27
  • I am putting it on the `def export():` in `mylib.io`. – pir Sep 04 '15 at 09:28
  • I'm sure you've seen [*this*](http://stackoverflow.com/a/4299378/23771). It won't tell you the time used by *every* line, only the ones big enough to worry about. And, it's not just CPU time, but wall-clock time, which is important since you're doing I/O. – Mike Dunlavey Sep 04 '15 at 15:10

0 Answers0