37

I'm running a relatively complex python program and in it there is a montecarlo simulation which takes up most of the time. I would like to find out what part of it uses the most resources so I can potentially make it faster.

I'm using PyCharm Professional edition and tried to use the profiler, but the result is only a large list of irrelevant functions that I've never heard of.

Questions: Is there a good profiler I can use that delivers meaningful results so I can see which function or keyword uses the most resources in my montecarlo simulation?

Nickpick
  • 6,163
  • 16
  • 65
  • 116
  • 3
    Don't go very long without trying [*this*](http://stackoverflow.com/a/4299378/23771). It costs nothing and is surprisingly effective at telling you what takes time. – Mike Dunlavey Oct 05 '15 at 17:41

3 Answers3

28

Depending on your needs and your python version, maybe you want to use something like hotshot. https://docs.python.org/2/library/hotshot.html

EDIT:

For python 3.4 cProfile is probably one the best options you have available but you will definitely have to filter the results with grep/sed/awk to be able to get the relevant results especially if you use libraries imported where there are a lot of internal calls occurring.

I like sorting by number of calls: python -m cProfile -s 'calls' <your_program>.py

Now the issue in python3 with that method is the number of primitive calls that will show up if cProfile is called externally, so running it internally is probably a better idea:

import cProfile

pr = cProfile.Profile()
pr.enable()
your_function_call()
pr.disable()
# after your program ends
pr.print_stats(sort="calls")
shafeen
  • 2,431
  • 18
  • 23
28

Note: As mentioned in the comments, the following applies to the paid version of PyCharm:

If using 3.x (don't know about 2.x), I'll add to shafeen's answer and make it more PyCharm specific as per the original post. This also works better for web applications or larger applications versus simple command line programs where printing the output to stdout might be okay (still better to be able to sort different ways through PyCharm's viewer).

Indeed, do as suggested by instantiating Profile and enabling and disabling as needed. To make that useful though, you'll want to save this to a file.

  • In an outer section of your code, instantiate Profile.
  • In the inner section of your code, do your profiling.
  • Now, call pr.dump_stats('profile.pstat')

You now have a profile file that you would like to examine. Go to Tools|Open CProfile snapshot. Select profile.pstat and now you can view and sort by different headings as desired.

Summary

import cProfile as profile

# In outer section of code
pr = profile.Profile()
pr.disable()

# In section you want to profile
pr.enable()
# code of interest
pr.disable()

# Back in outer section of code
pr.dump_stats('profile.pstat')

Open file in PyCharm's CProfile viewer.

Software Prophets
  • 2,838
  • 3
  • 21
  • 21
  • 14
    Unfortunately the profiler plugin is in the Professional (non-free) edition of PyCharm. – P-Gn Aug 03 '17 at 08:10
  • 3
    How is this different than using the built in profiler? https://www.jetbrains.com/help/pycharm/profiler.html – Josh Hibschman May 31 '18 at 15:29
  • 12
    @y3sh The built in profiler runs for the entire time whereas placing profile code at certain points, invoking enable/disable, permits you to zoom in on a specific area of code you want to look at. Wading through all of the output of a full run can be laborious and there may be a lot more noise than you (I) might want to deal with when your area of interest is already known. You might start with the run configuration and then switch to what I show once you've found a trouble area? The OP explains this very difficulty - too much noise. – Software Prophets May 31 '18 at 19:17
  • Does this provide information about the call tree leading to the functions where lots of time is spent, to clarify where the low level calls come from, and so you know enough about the call tree to make sense of where to optimize things? cf https://stackoverflow.com/questions/4544784/how-can-you-get-the-call-tree-with-python-profilers – nealmcb Jul 21 '19 at 20:58
  • opening in PyCharm CProfile viewer: https://stackoverflow.com/questions/33906456/use-pycharm-to-view-profiling-data – marke Jun 22 '22 at 12:20
  • Intellij idea - opening dumped .pstat file displays 'no profiler data'. File itself has some generated (binary) content, also printing stats to console seems to work ok – stam May 20 '23 at 17:00
8

I am surprised nobody mentioned SnakeViz yet. Besides the profiler from Spyder this is the best python profiler I could find so far:

pip install snakeviz

Then:

python -m cProfile -o program.prof my_program.py
snakeviz program.prof

It will open some nice visualizations in your browser:

SnakeViz demo

Ohumeronen
  • 1,769
  • 2
  • 14
  • 28