Profile Python import times

Question

Is there a way to find out which imports are taking the longest in Python? Looking at the output of python -m cProfile <script>, it doesn't seem to include import statements (understandably given potentially huge dependency trees). Initially I thought it did since I saw a row for __import__() calls, but I think this might actually be because code somewhere is explicitly calling it, toy programs with only import statements don't have a row for it.

Right now I'm just using:

start = time.time()
import <module>
print '%s / time: %f' % (<module>, time.time()-start)

around each module, but it doesn't profile it recursively to see which import within an import might be inflating the time.

Why would you need to profile the imports? They only happen once, and either you need them or you don't. — jonrsharpe, Jan 12 '16 at 23:30
If you're looking for bottlenecks in the import system, you're looking at the wrong place. — Dimitris Fasarakis Hilliard, Jan 12 '16 at 23:34
To add to the other comments, if you're seeing imports that take time, it's because you have code in those files that isn't inside of a class/function definition and not guarded by `if name == '__main__'` Unless there's some strict need for it, you could put this code in something like `init()` and profile that. Not guarding this code will also result in the code being run every time that file is imported. — Daniel Underwood, Jan 12 '16 at 23:38
@danielu13 That was the intention of my question, to find out which imports had a lot of initialization going on (one is taking 22 seconds). While not at all significant to the total runtime, I am curious where that time is coming from. — Andrew Nguyen, Jan 14 '16 at 17:59
Why not refactor your modules to have an `init()` function or something similar that calls the code that is running on import? This would allow you to profile that function and prevent the hazard of importing the same module multiple times if you had something like `a.py` that has `import b; import c` while `b.py` also has an `import c`. — Daniel Underwood, Jan 14 '16 at 18:20
It's a large codebase with a large dependency tree so just finding the initialization code to refactor would be difficult. You don't have to worry about repeated imports, Python only initializes an import once (http://stackoverflow.com/a/296062/873472). — Andrew Nguyen, Jan 14 '16 at 19:15
It makes sense to reduce import time e.g. in cli apps, where it can be majority of the running time. — Artimi, Apr 25 '23 at 08:24

score 51 · Accepted Answer · edited Oct 20 '21 at 21:58

51

This is a totally legitimate question. For instance, it makes sense to try to accelerate cold start for CLI applications. Python 3.7 now offers an option to print the import time: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPROFILEIMPORTTIME

You can either run:

python -X importtime myscript.py

or:

PYTHONPROFILEIMPORTTIME=1 myscript.py

EDIT: to view those results I recommend tuna.

edited Oct 20 '21 at 21:58

sth

222,467
53
283
367

answered Apr 10 '19 at 01:26

Régis B.

10,092
6
54
90

2

What units are those numbers in? It's not mentioned in the docs. – exhuma Oct 05 '20 at 06:24
In python 3.10, at least, the column header shows [us], i.e. microseconds. – Chris L. Barnes Oct 03 '22 at 14:57

score 1 · Answer 2 · answered Mar 10 '16 at 15:11

As stated in danielu13's comment, what you really want to profile is the code executed inside a module upon import of this module.

It seems that cProfile includes this code executed upon imports in its output. It shows up as <module> for module level code. Many packages are imported as __init__.py files, so gprof2dot shows __init__:23:<module> (Line numbers may differ), which does not tell you which package the file comes from.

Using cprofilev, you can find out what files (and thus packages) take most time.

One sidenote: Profiling imports actually can make sense in some cases. On my system, importing the module networkx (https://networkx.github.io/) takes 1.7 seconds.

score 1 · Answer 3 · answered Aug 08 '19 at 08:06

1

A very simplistic and not hierarchical solution for Python 2.7 that prints module name and import time since last module:

LAST_TIME = time.time()

class ImportHook(object):
    def find_module(self, fullname, path=None):
        global LAST_TIME
        cur_time = time.time()
        delta = cur_time - LAST_TIME
        long_time = '!!!!!!!!!!!!!!\n' * 10 if delta > 0.05 else ''
        print '%0.3f %s \n %s' % (delta, long_time, fullname),
        LAST_TIME = cur_time

import sys
sys.meta_path.insert(0, ImportHook())

answered Aug 08 '19 at 08:06

Yariv

381
3
11

I would love to understand why that timer works. Is that the actual import time, or just the elapsed time since the last time the meta_path was searched for a module? – Ryan de Kleer Jun 02 '21 at 05:07
@RyandeKleer , just the delta from last search. – Yariv Jun 03 '21 at 09:07

Profile Python import times

3 Answers3