Grouping Parametrized Benchmarks with pytest

Question

I'm currently benchmarking an implementation of an AVL Tree I made against a non-rebalancing binary search tree using pytest-benchmark. It seems to be working well for me so far but I've run into an issue. In order to consolidate the test file, I found that I can parametrize tests and also group the output of the benchmark for readability but I can't seem to do them both at the same time.

My current insertion benchmark:

# always the same for repeatability
random.seed(0x1C2C6D66)

def insertRandomOrder(t, n):
  tree = t()
  for i in range(0,n):
    tree.insert(random.randint(0,0x7FFFFFFF),i)

def insertDescendingOrder(t, n):
  tree = t()
  for i in range(0,n):
    tree.insert(n-i,i)

def insertOutInOrder(t, n):
  tree = t()
  for i in range(0,n):
    idx = (i%2)*n + (1-2*(i%2))*i
    tree.insert(idx,i)

def insertAscendingOrder(t, n):
  tree = t()
  for i in range(0,n):
    tree.insert(i,i)

types = [BaseTree, AvlTree]
sizes = [100,300,1000]
cases = [insertAscendingOrder, insertDescendingOrder, insertOutInOrder, insertRandomOrder]

@pytest.mark.parametrize('t', types)
@pytest.mark.parametrize('n', sizes)
@pytest.mark.parametrize('case', cases)
def test_insert_benchmark(benchmark, t, n, case):
  benchmark(case, t, n)

And here is the output:

Does anyone know of a way where I could group the output but say, the case from my example? Or better yet, by a (case,n) tuple?

There is support for specific param grouping in the master branch. Just `pip install https://github.com/ionelmc/pytest-benchmark/archive/master.zip` and then use `--benchmark-group-by=param:case`. I think we could support the `case,n` situation ... — ionelmc, Feb 02 '16 at 10:03
I did see the param option, but didn't know about the `:` option in the main branch. I get an unrelated error in the master branch, though when I try to run the benchmark. "File "xxx\python35\lib\site-packages\pytest_benchmark\storage.py line 41 `if candidate.is_file():`", "OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '[0-9][0-9][0-9][0-9]_*'" (running on Windows 10... work computer. I'll try again when I get home on my preferred Linux Dev Environment) — Frank Bryce, Feb 02 '16 at 17:23
FYI, I've fixed the Windows issue and you can have `--benchmark-group-by=param:foo,param:bar` now (in master branch) — ionelmc, Feb 04 '16 at 23:33

score 1 · Accepted Answer · answered Feb 02 '16 at 18:38

There was a useful comment saying that the master branch of pytest is in the process of supporting this exact feature, but I was unable to get it to work (fingers crossed for next release).

In the meantime, I figured out this handy work around. I'm able to group by case, but not by (case,n) with this method. I added a @benchmark_this decorator above each test case to wrap the benchmark call. It's pretty handy even without the extra benefit of grouping by test case!

def benchmark_this(test):
  def wrapper(benchmark, t, n):
    benchmark(test, None, t, n)
  return wrapper

types = [BaseTree, AvlTree]
sizes = [100,300,1000]

@pytest.mark.parametrize('t', types)
@pytest.mark.parametrize('n', sizes)
@benchmark_this
def test_insertRandomOrder(benchmark, t, n):
  random.seed(0x1C2C6D66)
  tree = t()
  for i in range(n):
    tree.insert(random.randint(0, 0x7FFFFFFF), i)

@pytest.mark.parametrize('t', types)
@pytest.mark.parametrize('n', sizes)
@benchmark_this
def test_insertDescendingOrder(benchmark, t, n):
  tree = t()
  for i in range(n):
    tree.insert(n-i, i)

# ...

Invoked with

py.test --benchmark-group-by=func

Grouping Parametrized Benchmarks with pytest

1 Answers1

Linked