3

I write most of my codes as Fortran extension to Python, using fantastic f2py tool. However, it is sometimes difficult to catch the memory leaks, and profile the program (where the most time it is spent).

The question is: Is there some simple way to debug and/or profile extensions (especially, f2py-generated) for Python? Using valgrind seems to be complicated (I use MacOsX, and do not want to recompile the interpreter). The only options that I have now is print - debugging + print - profile, which is time-consuming.

Ivan Oseledets
  • 2,270
  • 4
  • 23
  • 28
  • 1
    Have you tried python's `profile` or `cProfile` modules? As far as I understand it, they instrument your code for you and should provide reasonably accurate timings of everything in python. (if `foo` is a fortran subroutine, you won't be able to get information about what `foo` calls). – mgilson Jul 17 '12 at 19:06
  • As far as memory leaks are concerned, you should just write simple test routines in fortran which call your fortran routines. Those can be run under valgrid without any problem. – mgilson Jul 17 '12 at 20:13
  • @mgilson , profile & cProfile can allow me to see the actual running times for the modules. Using valgrind with python seems to be complicated http://stackoverflow.com/questions/3982036/how-can-i-use-valgrind-with-python-c-extensions – Ivan Oseledets Jul 18 '12 at 09:21
  • 2
    I wasn't trying to say that you should use valgrind with python -- only with the fortran code. If you have a fortran subroutine `foo`, you can write a simple fortran code to use `foo`. Compile that whole thing and use valgrind on it. If it doesn't leak, then `foo` doesn't leak either and you can use it safely with python. – mgilson Jul 18 '12 at 12:27
  • @IvanOseledets, I use valgrind's callgrind tool to profile my f2py extension, and it's very simple. The question you refer to discusses difficulties with the memcache tool, so maybe it's not so easy to detect memory leaks. But profiling is definitely straightforward. – DaveP Jul 23 '12 at 01:37
  • @DaveP Can you be more elaborate on the details how do you use valgrind? – Ivan Oseledets Jul 24 '12 at 07:59
  • 1
    @IvanOseledets: The basic operation is to run "valgrind --tool=callgrind python myscript.py", (where myscript.py calls your f2py module) then examine the output using the convenient kcachegrind GUI. Make sure that you choose your script to run fairly quickly under normal execution, as running under valgrind is at least 10x slower. – DaveP Jul 24 '12 at 22:52

1 Answers1

2

In case any others have the same problem but on Linux platforms (sorry, OS/X is not supported so I know this is not "the Answer" for you) - the Allinea tools can profile and debug the Fortran extensions called in Python, and see any memory leaks.

Compared to callgrind/kcachegrind, the profiler runs much faster (having max 5% slowdown typically) and it's much deeper as to why the code runs slow as it understands vectorization and running in real time means I/O profiling is accurate.

There's a blog exploring f2py extension debugging and profiling that introduces it.

David
  • 756
  • 5
  • 10