73

Sometimes, I like to time how long it takes parts of my code to run. I've checked a lot of online sites and have seen, at large, two main ways to do this. One is using time.time and the other is using timeit.timeit.

So, I wrote a very simple script to compare the two:

from timeit import timeit
from time import time
start = time()
for i in range(100): print('ABC')
print(time()-start, timeit("for i in range(100): print('ABC')", number=1))

Basically, it times how long it takes to print "ABC" 100 times in a for-loop. The number on the left is the results for time.time and the number on the right is for timeit.timeit:

# First run
0.0 0.012654680972022981
# Second run
0.031000137329101562 0.012747430190149865
# Another run
0.0 0.011262325239660349
# Another run
0.016000032424926758 0.012740166697164025
# Another run
0.016000032424926758 0.0440628627381413

As you can see, sometimes, time.time is faster and sometimes it's slower. Which is the better way (more accurate)?

  • 8
    `timeit` is the better choice for timing chunks of code. It uses `time.time()` (`time.clock()` for Windows) and disables the garbage collector. Also, one trial isn't really enough. – Blender Jul 10 '13 at 19:34
  • 5
    @Blender: `timeit` uses `time.perf_counter` in Python 3.3+ – jfs Dec 11 '13 at 21:37

2 Answers2

97

timeit is more accurate, for three reasons:

  • it repeats the tests many times to eliminate the influence of other tasks on your machine, such as disk flushing and OS scheduling.
  • it disables the garbage collector to prevent that process from skewing the results by scheduling a collection run at an inopportune moment.
  • it picks the most accurate timer for your OS, time.time or time.clock in Python 2 and time.perf_counter() on Python 3. See timeit.default_timer.
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • 1
    If you explicitly do those three things you get results as accurate as timeit? My use case is running many tests with different inputs, so figure it might be easier to dispense with timeit. – AnnanFay Apr 17 '14 at 01:01
  • 9
    @Annan: Why reinvent this wheel? I'm sure that whatever usecase you can come up with, provided you understand what it means to run something repeatedly, can be supplied for by `timeit`. There are a few smaller minor tricks `timeit` uses (such as using `itertools.repeat(None, repetitioncount)` for a low-friction repetition range) that you'd have to replicate too if you want to be as 'accurate' as `timeit` gets. – Martijn Pieters Apr 18 '14 at 09:52
  • 1
    @MartijnPieters: I'm wondering if `%timeit` is the built-in magic command for `timeit`, how about `%time`. Is `%time` the same when we use `time.clock()` (put 2 time.clock at the beginning and the end of the code and calculate the time difference). I've read here but not enough info https://ipython.readthedocs.io/en/stable/interactive/magics.html . Thanks – Chau Pham Oct 13 '18 at 08:14
  • @Catbuilts: `%time` doesn't do any repetition. It's a straightforward 'run once, report time differences' operation. – Martijn Pieters Oct 13 '18 at 08:30
  • @MartijnPieters: Sorry if my question was not clear, I understand `%time` doesn't do repetition, while `%timeit` does. My question was: Is `%time` a shortcut (built-in magic command) to call `time.clock()` ? From the link ipython.readthedocs.io/en/stable/interactive/magics.html, it said that `%timeit` is Time execution of a Python statement using the `timeit` module, but it didn't tell anything about `%time`. I wonder where `%time` comes from, what exactly runs under the hood. Thanks – Chau Pham Oct 13 '18 at 08:40
  • 2
    @Catbuilts: it's a [bit more complicated than that](https://github.com/ipython/ipython/blob/0fbad83245f474c4dd7a1b1fbe7f2ea79e005653/IPython/core/magics/execution.py#L1226-L1244), but not much. – Martijn Pieters Oct 13 '18 at 08:43
  • @MartijnPieters: Thank you for the link. It uses `time.time()` to calculate `wall time`. – Chau Pham Oct 13 '18 at 08:56
  • But `%% timeit` uses first statement as the 'set up code'. What the hell that is supposed to mean? – amsquareb Apr 02 '20 at 08:05
  • 1
    @amsquareb: The setup block is run *once*, letting you create the right objects (context) for the repeated tests. – Martijn Pieters Apr 02 '20 at 14:09
  • I'm not convinced that disabling the garbage collector always makes it more accurate. See [this case](https://stackoverflow.com/q/63757763/13008439) where I think the garbage collection is responsible for most of the time and it is caused by the measured code so it shouldn't be ignored. – superb rain Sep 05 '20 at 20:00
  • @superbrain interesting; adding `gc.disable()` and `gc.enable()` around your second timed test indeed eliminates the difference. This requires further investigation as there are no circular references there for the gc to bother with. You appear to have hit a rare edge case. – Martijn Pieters Sep 06 '20 at 02:07
6

At any given time, the Central Processing Unit (CPU) is used and shared by many processes. Measurements taken using time.time are relative to what we call wall clock. This means that the results are dependent to the other processes that were running at the time the test was executed. Therefore, in many cases the results produced by time.time are not as accurate as possible.

More reliable results can be generated using time.clock for Python 2.x and time.process_time() or time.perf_counter() for Python 3.X, that measures the CPU cycles used during the execution of the code but even this method as it heavily relies on the specific machine you are executing the tests. For example, the results might hugely differ if the tests are executed on different machines (even though the algorithm and input data are both exactly the same)

timeit.timeit is a an advanced library that is more accurate and reliable compared to time.time and time.clock as it takes into account the factors that pose the variance among the code executions and trials, by simply repeating the execution of tests in order to produce more reliable and accurate results.+

Giorgos Myrianthous
  • 36,235
  • 20
  • 134
  • 156