What is the logic behind the design for timeit?

Question

I am new to Python and figured I'd play around with problems on Project Euler to have something concrete to do meanwhile.

I came across the idea of timing different solutions to see how they rate against each other. That simple task turned out to be too complicated for my taste however. I read that the time.clock() calls are not accurate enough on unix systems (seconds resolution is simply pathetic with modern processors). Thus I stumbled upon the timeit module which seems to be the first choice for profiling tasks.

I have to say I really don't understand why they went with such a counter-intuitive way to go about it. I can't seem to get it to work, without needing to rewrite/restructure my code, which I find very frustrating.

Take the code below and nevermind for a second that it's neither pretty nor particularly efficient:

import math
import sys
from timeit import Timer

def digitsum(number):
    rem = 0 
    while number > 0:
        rem += number % 10
        number //= 10
    return rem

def prime_form(p):
    if p == 2 or p == 3 or p == 5:
        return True
    elif (p-1) % 6 != 0 and (p+1) % 6 != 0:
        return False
    elif digitsum(p) % 3 == 0: 
        return False
    elif p % 10 == 0 or p % 10 == 5:
        return False
    else:
        return True

def lfactor(n):

    if n <= 3:
        return 1

    limit = int(math.sqrt(n))
    if limit % 2 == 0:
        limit -= 1

    lfac = 1
    for i in range(3,limit+1,2):
        if prime_form(i):
            (div,rem) = divmod(n,i)
            if rem == 0:
                lfac = max(lfac, max(lfactor(div) ,lfactor(i)))

    return lfac if lfac != 1 else n

number = int(sys.argv[1])
t = Timer("""print lfactor(number)""", """import primefacs""")
t.timeit(100)
#print lfactor(number)

If i would like to time the line print lfactor(number) why should I go through a bunch of loops, trying to define a setup statement etc.. I understand why one would want to have debug tool that are detached from the code being tested (a la unit testing) but shouldn't there be a simple and straightforward way to get the process time of a chunk of code without much hassle (importing/defining a setup etc)? What I am thinking here is something like the way one would do that:

long t0 = System.currentTimeInMillis();
// do something
long t = System.currentTimeInMillis() - t0;

.. or even better with MATLAB, using the tic/toc commands:

tic
x = A\b;
t(n) = toc;

Hope this doesn't come across as a rant, I am really trying understand "the pythonian way of thinking" but honestly it doesn't come naturally here, not at all...

Answer 1

Simple, the logic behind the statement and setup is that the setup is not part of the code you want to benchmark. Normally a python module is loaded once while the functions inside it are run more than one, much more.

A pythonic way of use timeit?

$ python -m timeit -h

Tool for measuring execution time of small code snippets.

This module avoids a number of common traps for measuring execution
times.  See also Tim Peters' introduction to the Algorithms chapter in
the Python Cookbook, published by O'Reilly.

Library usage: see the Timer class.

Command line usage:
    python timeit.py [-n N] [-r N] [-s S] [-t] [-c] [-h] [--] [statement]

Options:
  -n/--number N: how many times to execute 'statement' (default: see below)
  -r/--repeat N: how many times to repeat the timer (default 3)
  -s/--setup S: statement to be executed once initially (default 'pass')
  -t/--time: use time.time() (default on Unix)
  -c/--clock: use time.clock() (default on Windows)
  -v/--verbose: print raw timing results; repeat for more digits precision
  -h/--help: print this usage message and exit
  --: separate options from statement, use when statement starts with -
  statement: statement to be timed (default 'pass')

[cut]


$ python -m timeit -s 'from primefacs import lfactor' 'lfactor(42)'

$ # this does not work, primefacs is not binded, ie. not loaded
$ python -m timeit 'primefacts.lfactor(42)'


$ # this does not work too, lfactor is not defined
$ python -m timeit 'lfactor(42)'

$ # this works but the time to import primefacs is benchmarked too
$ # but only the first time is loaded, the successive ones the cache is used. 
$ python -m timeit 'import primefacts; primefacts.lfactor(42)'

As you can see, the way timeit works is much more intuitive than you think.

Edit to add:

I read that the time.clock() calls are not accurate enough on unix systems (seconds resolution is simply pathetic with modern processors).

quoting the documentation:

On Unix, return the current processor time as a floating point number expressed in seconds. The precision, and in fact the very definition of the meaning of “processor time”, depends on that of the C function of the same name, but in any case, this is the function to use for benchmarking Python or timing algorithms... The resolution is typically better than one microsecond.

going on..

I have to say I really don't understand why they went with such a counter-intuitive way to go about it. I can't seem to get it to work, without needing to rewrite/restructure my code, which I find very frustrating.

Yes, it could be but then this is one of those cases where documentation can help you, here a link to the examples for the impatiens. Here a more gentle introduction to timeit.

Answer 2

When timing a statement, you want to time just that statement, not the setup. The setup could be considerably slower than the statement-under-test.

Note that timeit runs your statement thousands of times to get a reasonable average. It does this to eliminate the effects of OS scheduling and other processes (including but not limited to disk buffer flushing, cronjob execution, memory swapping, etc); only an average time would have any meaning when comparing different code alternatives.

For your case, just test lfactor(number) directly, and just use the timeit() function:

timeit.timeit('lfactor(number)', 'from __main__ import lfactor, number')

The setup code retrieves the lfactor() function, as well as number taken from sys.argv from the main script; the function and number won't otherwise be seen.

There is absolutely no point in performance testing the print statement, that's not what you are trying to time. Using timeit is not about seeing the result of the call, just the time it takes to run it. Since the code-under-test is run thousands of times, all you'd get is thousands of prints of (presumably) the same result.

Note that usually timeit is used to compare performance characteristics of short python snippets; to find performance bottlenecks in more complex code, use profiling instead.

If you want to time just one run, use the timeit.default_timer() function to get the most accurate timer for your platform:

timer = timeit.default_timer
start = timer()
print lfactor(number)
time_taken = timer() - start