4

What mechanisms does IPython provide for profiling code and scripts?

I found this interesting thread (March 2013) that suggests defining custom magics that wrap calls to other Python modules: line-profiler, psutil, memory_profiler.

However, looking at the GitHub discussions, I also found this issue (already closed) where Fernando, tk and other core IPython developers discuss the inclusion of magics by default in IPython for profiling.

All that said I could not find any information about profiling in the official dev documentation.

I am currently using the master branch of IPython. How can I profile a given script e.g. foo.py either line-by-line or by function call?

Amelio Vazquez-Reina
  • 91,494
  • 132
  • 359
  • 564
  • What's the goal? If the goal is to do profiling "because it's there", you will find something or other. OTOH if the goal is to speed up some code, [*this might be helpful*](http://stackoverflow.com/a/4299378/23771). – Mike Dunlavey Aug 07 '13 at 02:21
  • @MikeDunlavey - I am not sure I understand your question. There are many things that you can learn from profiling your code, e.g. gain insight of where the code spend most of its time, is it I/O? doing the mathematical processing of my large numpy arrays? This information can help me redefine the goal/scope of what I want my program to accomplish, the HW architectures that I should target, etc. The fundamental problem is the same though: Gain insight by profiling my code line-by-line and by function call, and to the extent possible, get a sense of how much memory my data structures are using. – Amelio Vazquez-Reina Aug 07 '13 at 17:28
  • For memory profiling I don't have any answers. For performance profiling, what I see is it's a favorite subject to talk about, but very little actual results in terms of speeding up code. The typical experience people seem to have is: The profiler gave me lots of numbers, but I went and looked at the "hot" code and didn't see much to fix, so since profilers are so highly respected, there must be nothing *to* fix (except for little stuff). The problem is not so much in the data they collect. The problem is the presentation cannot show enough context so you really understand. – Mike Dunlavey Aug 07 '13 at 19:31
  • Some profilers are better than others. I think the best ones are the ones that take stack samples, on wall-clock time (so as not to be blind to I/O), and that report percent (not absolute) time by line-of-code (not just function), and give you a butterfly view of those. Zoom from RotateRight is an example. But it would be even better if they let you examine actual stack samples, so you can see the full reason for what's happening. You could only examine a small number of them, but oddly enough, that's statistically sufficient. – Mike Dunlavey Aug 07 '13 at 19:47
  • It's a matter of quality versus quantity of samples. If you have an infinite loop, how many samples do you need to find it? Answer: one sample of the program's state, examined very carefully, because you know you're in the loop. If the loop is not infinite, just very long-running, how many samples do you need? If the activity, whatever it is, consumes fraction _f_ of the time, the number of samples you need to see it twice is, on average, _2/f_, and twice is all you need to know it's significant. – Mike Dunlavey Aug 08 '13 at 16:18

0 Answers0