Pytest-benchmark: precomputing test data

Question

Could you tell me please if precomputation of test input data in the same script is a valid approach, and if that would not affect the benchmarking procedure (i.e. timing)?

E.g., in script.py

def compute_inputs():
    #some computations
    #return inputs

INPUTS = compute_inputs()

def something(some_arg):
    # code to be benchmarked

def test_my_stuff(benchmark):
    # benchmark something
    result = benchmark(something)

And then run it like this:

pytest script.py

Thank you for any input.

Max Skoryk · Accepted Answer · 2022-09-27T09:19:44.480

TLDR; According to the pytest-benchmark documentation it only benchmarks a callable that you pass to it. So in theory, any variables you've precomputed before the call to the function you benchmark, will not affect the result (timing).

We can make an example and see it in practice:

def something():
    time.sleep(1)
    return 123


def test_my_stuff(benchmark):
    result = benchmark(something)

    assert result == 123

The example above sleeps for 1 second, so we'd expect the runtime to be about that. And it's runtime is indeed ~1 second:

------------------------------------------- benchmark: 1 tests ------------------------------------------
Name (time in s)        Min     Max    Mean  StdDev  Median     IQR  Outliers     OPS  Rounds  Iterations
---------------------------------------------------------------------------------------------------------
test_my_stuff        1.0005  1.0050  1.0031  0.0019  1.0028  0.0032       1;0  0.9969       5           1
---------------------------------------------------------------------------------------------------------

Now let's try a case, where we precompute some values:

def compute_inputs():
    time.sleep(1)
    return 123

INPUT = compute_inputs()

def something():
    time.sleep(1)
    return INPUT


def test_my_stuff(benchmark):
    result = benchmark(something)

    assert result == 123

Here, we can treat INPUT as a precomputed input. It takes one second to produce INPUT, so we could expect the total runtime to be 2 seconds. But as you can see in the output, pytest-benchmark ignores this precomputation and only counts the callables that are executed inside the benchmark fixture:

------------------------------------------- benchmark: 1 tests ------------------------------------------
Name (time in s)        Min     Max    Mean  StdDev  Median     IQR  Outliers     OPS  Rounds  Iterations
---------------------------------------------------------------------------------------------------------
test_my_stuff        1.0008  1.0050  1.0030  0.0019  1.0038  0.0033       2;0  0.9970       5           1
---------------------------------------------------------------------------------------------------------

Pytest-benchmark: precomputing test data

1 Answers1