How to see what is slowing your Python script down using snakeviz and cProfile

Question

So you want to know why your Python script is running slow?

There is an easy way to do this using cProfile and pstats, which count the number of calls to each function. We can visualize the results interactively in our web browser using snakeviz. You'll need to pip install snakeviz but the other two are included in the default Python libraries.

I originally had been searching for a way to do all this inside a single Python script within PyCharm, but after finding only solutions that involved moving over to the terminal to run snakeviz, I managed to solve this myself by looking through the snakeviz source code. I thought it would be beneficial to the community if I shared my recent discovery. See below for details of how to create a simple wrapper function and pass it across for speed testing.

score 0 · Answer 1 · answered Jul 02 '21 at 01:37

The following code shows how to speed test your script by placing it in a wrapper and passing that wrapper for analysis. When it is finished, the function should automatically open a new tab in your browser to display the results.

# this function does all the magic
# it accepts any function you want to speed test
# then it performs the test, analyses the results, and provides the output in your browser
def speedtest(function_wrapper):
    import cProfile
    import pstats
    import snakeviz.cli as cli

    with cProfile.Profile() as pr:
        function_wrapper()
    stats = pstats.Stats(pr)
    stats.sort_stats(pstats.SortKey.TIME)
    filename = "speedtest_profile.prof"
    stats.dump_stats(filename=filename)
    cli.main([filename])


# simply place the code you want to speed test inside a wrapper function
# it can be called anything but wrapper is used here
def wrapper():
    from reliability.Fitters import Fit_Weibull_2P
    from reliability.Distributions import Weibull_Distribution

    data = Weibull_Distribution(alpha=50, beta=2).random_samples(1000000, seed=1)
    Fit_Weibull_2P(failures=data, show_probability_plot=False, print_results=False)


# this executes the speedtest on the wrapper function
speedtest(wrapper)

The results look like this:

From this I can clearly see that I should work on speeding up the Probability_plotting.plotting_positions function. In case you're wondering, the function I have wrapped is the generation of 1 million samples from a Weibull Distribution, and then fitting a 2 parameter Weibull distribution to those samples. I wanted to see what parts of this process took the longest.

How to see what is slowing your Python script down using snakeviz and cProfile

1 Answers1