3

CPython automatically compiles and caches all .py files into .pyc files in the same directory or a __pycache__ directory, depending on the version. How much speed improvement does using these cached files provide?

I an not asking about the execution speed of the python code - that question has been answered many times: "there is no difference in execution speed, the .pyc files are only to improve the startup speed."

I am asking how much that improvement in startup speed is.

evil otto
  • 10,348
  • 25
  • 38
  • 2
    A good computer scientist would say "it depends...". I think this question is probably too broad. – Peter Wood Mar 12 '18 at 22:56
  • You can use the [py_compile](https://docs.python.org/3/library/py_compile.html) module to do this yourself, like [this](https://stackoverflow.com/a/2042442/9348376), but it really depends on your program - It saves you however long it would have taken to compile. – Sean Breckenridge Mar 12 '18 at 22:59

1 Answers1

0

It depends on how complex the modules in question are, how many you're importing, etc.

In local tests, I copied collections.py to my local directory, then tested with:

time python -B -E -S -c 'import collections; print(collections)'

to try to nail down the rough end-to-end cost of the collections module alone without cached bytecode, then the same without -B (so it would create and use bytecode caches). The difference was around 5 ms (36 ms with -B, then 31 ms without -B on second and subsequent runs).

For more fine-grained testing, explicitly compiling without invoking any other import machinery using ipython %timeit magic got:

>>> with open('collections.py') as f: data = f.read()
...

>>> %timeit -r5 compile(data, 'collections.py', 'exec', 0, 1)
100 loops, best of 5: 2.9 ms per loop

That's omitting some of the other import machinery work, just recompiling over and over, and it runs ~3 ms, which seems reasonable. If you're importing a hundred source modules (not entirely unreasonable, counting all the cascading imports the handful of explicit imports trigger), saving 1-5 ms each can make a meaningful difference for short program runs.

ShadowRanger
  • 143,180
  • 12
  • 188
  • 271
  • Note: To be clear, my test was done with Python 2, but I wouldn't expect significantly different results on Python 3. I followed up with the Python 3.5 equivalent (copying the whole `collections` directory, since it's not a single file anymore), and the difference was larger, around 10 ms less (from 51-52 down to 39-41 ms), though that could be due to the more complex package design (more cascading imports occur). The `compile` microbenchmark also took longer, around 8 ms rather than 3 ms. – ShadowRanger Mar 12 '18 at 23:14