Benchee warmup - how does it work in this library?

Question

I'm playing with the warmup property of Benchee benchmarks increasing the warmup time gradually. I was expecting to get better results with a longer warmup time, but I get the opposite results.

For example, running the following example (extracted from the Benchee docs):

list = Enum.to_list(1..10_000)
map_fun = fn i -> [i, i * i] end

Benchee.run(
  %{"flat_map" => fn -> Enum.flat_map(list, map_fun) end},
  warmup: 0,
)

Benchee.run(
  %{"flat_map" => fn -> Enum.flat_map(list, map_fun) end},
  warmup: 2,
)

Benchee.run(
  %{"flat_map" => fn -> Enum.flat_map(list, map_fun) end},
  warmup: 4,
)

Benchee.run(
  %{"flat_map" => fn -> Enum.flat_map(list, map_fun) end},
  warmup: 8,
)

Benchee.run(
  %{"flat_map" => fn -> Enum.flat_map(list, map_fun) end},
  warmup: 16,
)

I did several invocations of the above script and I got similar results. The first execution with warmup: 0 was the best option.

Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.10.3
Erlang 23.0.2

Benchmark suite executing with the following configuration:
warmup: 0 ns
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 5 s

Benchmarking flat_map...

Name               ips        average  deviation         median         99th %
flat_map        1.70 K      588.45 μs    ±14.33%         563 μs     1017.17 μs
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.10.3
Erlang 23.0.2

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 7 s

Benchmarking flat_map...

Name               ips        average  deviation         median         99th %
flat_map        1.67 K      600.24 μs    ±18.69%         563 μs     1085.84 μs
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.10.3
Erlang 23.0.2

Benchmark suite executing with the following configuration:
warmup: 4 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 9 s

Benchmarking flat_map...

Name               ips        average  deviation         median         99th %
flat_map        1.66 K      602.44 μs    ±18.32%         564 μs     1085.14 μs
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.10.3
Erlang 23.0.2

Benchmark suite executing with the following configuration:
warmup: 8 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 13 s

Benchmarking flat_map...

Name               ips        average  deviation         median         99th %
flat_map        1.65 K      606.06 μs    ±17.35%      573.98 μs     1072.98 μs
Operating System: macOS
CPU Information: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz
Number of Available Cores: 16
Available memory: 32 GB
Elixir 1.10.3
Erlang 23.0.2

Benchmark suite executing with the following configuration:
warmup: 16 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 21 s

Benchmarking flat_map...

Name               ips        average  deviation         median         99th %
flat_map        1.66 K      601.32 μs    ±17.79%         573 μs        1081 μs

In other VMs usually, you get better performance after a warmup phase.

How does the warmup work in BEAM? and particularly in Benchee?

Thanks in advance, Humberto

Answer 1

warmup in benchee is doing nothing more then just run your measured function for several amount of time without counting the results.

benchee creators point us out, that warmup concept is taken for JIT languages, where first code run will compile it and cache for future calls.

How the warmup can affect the results of the benchmark in BEAM? It's a question that absolutely impossible to answer, but let's try to find any caching mechanisms in it:

1. Atom's table can be populated during first function run, if it extensively uses atoms OR generates it on the fly.
2. May be the code itself (business logic) have some caching mechanism, that can somehow affect a benchmark.
3. File descriptors can be opened, if the benchmarked code uses IO that doesn't close the descriptors - for ex. files, http requests, etc. Also, Page cache of the underlinge OS (better to say File System) can be affected.

If your code is straightforward, dumb and Thread|CPU bounded, warmup will just add some degrees to your CPU and make your cooling system to work harder, without showing any sanity upon the results.