How to create randomized inputs for Criterion benchmarks in Rust

Question

I am trying to benchmark the annotate routine using the Criterion benchmarking library. The routine is checking a &[&str] parameter (2D square string) and returns Vec<String> and I suspect that its execution time might depend on the contents of the parameter. Hence, I want to randomize the inputs for a single workload, in this case, workload means width and height of the string parameter (usually width == height).

I noticed that Criterion has iter_batched and iter_batched_ref methods that take two closures instead of single timed closure, a Setup, and a Routine (timed). Both of them are FnMut so all the captured variables are mutable references (as far as I understand). So I tried to randomize the contents of a Vec<Vec<u8>> in the Setup while Routine is only calling my annotate function.

fn plant_mines<R: Rng + ?Sized>(mfield: &mut Vec<Vec<u8>>, rng: &mut R) {
    mfield.into_iter().flatten().for_each(|x| *x = if rng.gen::<f32>() < MINE_RATE {MINE} else {EMPTY});
}

pub fn benchmark(c: &mut Criterion) {
    let mut rng = rand_pcg::Pcg64Mcg::seed_from_u64(RAND_SEED);
    let bench_params : [(usize, usize); 4] = [
        (3, 3),
        (5, 5),
        (8, 8),
        (16, 16)
    ];

    let dims2str = |d : (usize, usize)| { let (w, h) = d; format!("{:}, {:}", w, h) };

    let mut group = c.benchmark_group("Minefield Benchmark");
    for dims in bench_params.iter() {
        group.bench_with_input(BenchmarkId::from_parameter(dims2str(*dims)), dims, |b, (w, h)| {
            let mut mf : Vec<Vec<u8>> = (0..*h).map(|_| iter::repeat(EMPTY).take(*w).collect::<Vec<_>>()).collect();
            let mf_ref = mf.iter().map(|vu8| str::from_utf8(&vu8).unwrap()).collect::<Vec<&str>>();

            b.iter_batched(|| {
                plant_mines(&mut mf, &mut rng);
            }, |_| annotate(&mf_ref), BatchSize::SmallInput);
        });
    }
    group.finish();
}

Normally, the Setup function should produce input for the Routine to be used but in this case, I omit them. The mf_ref holds a reference vector for Vec<Vec<u8>> and I don't want to create/allocate this vector in the timed Routine because otherwise, it will create more noise for the benchmark. Unfortunately, the borrow checker is not happy.

I tried to create the input vector within the Setup closure but then it cannot return the reference vector to the timed Routine. I tried to understand if either of the Cell or RefCell can solve the problem but I couldn't grasp them enough to apply in here (if possible). The Cell was not possible since Vec doesn't implement the Copy trait, which I understood.

How can I satisfy the compiler while benchmarking only the annotate function and use randomized inputs for each batch of samples?

Answer 1

You're not using the batch mechanism as it is intended, and doing so will solve your conflicting borrow problem. Instead of the benchmarked routine closing over mf_ref, you should create mf in the setup closure and return it. That way, there's no sharing of a mutable reference to mf.

b.iter_batched(|| {
    let mut mf : Vec<Vec<u8>> = (0..*h).map(|_| iter::repeat(EMPTY).take(*w).collect::<Vec<_>>()).collect();
    plant_mines(&mut mf, &mut rng);
    mf
}, |mf| annotate(&mf), BatchSize::SmallInput);

Note that this also got rid of mf_ref's transformation to &str. This is not possible to implement (easily) here unless the strings are all &'static str (which might be possible to do if they're always a string corresponding to MINE or EMPTY, so that you can just have two corresponding string literals); you'll need to redesign your annotate so it works on an owned structure or one that refers to only &'static strs. (Or you could put it inside the benchmark closure next to annotate, but then you're measuring it.)

While of course you shouldn't reshape your code just for the sake of a benchmark, this will probably give you flexibility elsewhere, as long-lasting references are generally inconvenient. I can't advise you on exactly what to do since I don't know what annotate does or what the values of MINE and EMPTY are.