How can I create parameterized tests in Rust?

Question

I want to write test cases that depend on parameters. My test case should be executed for each parameter and I want to see whether it succeeds or fails for each parameter.

I'm used to writing things like that in Java:

@RunWith(Parameterized.class)
public class FibonacciTest {
    @Parameters
    public static Collection<Object[]> data() {
        return Arrays.asList(new Object[][] {     
                 { 0, 0 }, { 1, 1 }, { 2, 1 }, { 3, 2 }, { 4, 3 }, { 5, 5 }, { 6, 8 }  
           });
    }

    private int fInput;

    private int fExpected;

    public FibonacciTest(int input, int expected) {
        fInput= input;
        fExpected= expected;
    }

    @Test
    public void test() {
        assertEquals(fExpected, Fibonacci.compute(fInput));
    }
}

How can I achieve something similar with Rust? Simple test cases are working fine, but there are cases where they are not enough.

#[test]
fn it_works() {
    assert!(true);
}

Note: I want the parameters as flexible as possible, for example: Read them from a file, or use all files from a certain directory as input, etc. So a hardcoded macro might not be enough.

Answer 1

The built-in test framework does not support this; the most common approach used is to generate a test for each case using macros, like this:

macro_rules! fib_tests {
    ($($name:ident: $value:expr,)*) => {
    $(
        #[test]
        fn $name() {
            let (input, expected) = $value;
            assert_eq!(expected, fib(input));
        }
    )*
    }
}

fib_tests! {
    fib_0: (0, 0),
    fib_1: (1, 1),
    fib_2: (2, 1),
    fib_3: (3, 2),
    fib_4: (4, 3),
    fib_5: (5, 5),
    fib_6: (6, 8),
}

This produces individual tests with names fib_0, fib_1, &c.

Answer 2

My rstest crate mimics pytest syntax and provides a lot of flexibility. A Fibonacci example can be very neat:

use rstest::rstest;

#[rstest]
#[case(0, 0)]
#[case(1, 1)]
#[case(2, 1)]
#[case(3, 2)]
#[case(4, 3)]
#[case(5, 5)]
#[case(6, 8)]
fn fibonacci_test(#[case] input: u32, #[case] expected: u32) {
    assert_eq!(expected, fibonacci(input))
}

pub fn fibonacci(input: u32) -> u32 {
    match input {
        0 => 0,
        1 => 1,
        n => fibonacci(n - 2) + fibonacci(n - 1)
    }
}

Output:

/home/michele/.cargo/bin/cargo test
   Compiling fib_test v0.1.0 (file:///home/michele/learning/rust/fib_test)
    Finished dev [unoptimized + debuginfo] target(s) in 0.92s
     Running target/debug/deps/fib_test-56ca7b46190fda35

running 7 tests
test fibonacci_test::case_1 ... ok
test fibonacci_test::case_2 ... ok
test fibonacci_test::case_3 ... ok
test fibonacci_test::case_5 ... ok
test fibonacci_test::case_6 ... ok
test fibonacci_test::case_4 ... ok
test fibonacci_test::case_7 ... ok

test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Every case is run as a single test case.

The syntax is simple and neat and, if you need, you can use any Rust expression as the value in the case argument.

rstest also supports generics and pytest-like fixtures.

---

Don't forget to add rstest to dev-dependencies in Cargo.toml.

Answer 3

Probably not quite what you've asked for, but by using TestResult::discard with quickcheck you can test a function with a subset of a randomly generated input.

extern crate quickcheck;

use quickcheck::{TestResult, quickcheck};

fn fib(n: u32) -> u32 {
    match n {
        0 => 0,
        1 => 1,
        _ => fib(n - 1) + fib(n - 2),
    }
}

fn main() {
    fn prop(n: u32) -> TestResult {
        if n > 6 {
            TestResult::discard()
        } else {
            let x = fib(n);
            let y = fib(n + 1);
            let z = fib(n + 2);
            let ow_is_ow = n != 0 || x == 0;
            let one_is_one = n != 1 || x == 1;
            TestResult::from_bool(x + y == z && ow_is_ow && one_is_one)
        }
    }
    quickcheck(prop as fn(u32) -> TestResult);
}

I took the Fibonacci test from this Quickcheck tutorial.

---

P.S. And of course, even without macros and quickcheck you still can include the parameters in the test. "Keep it simple".

#[test]
fn test_fib() {
    for &(x, y) in [(0, 0), (1, 1), (2, 1), (3, 2), (4, 3), (5, 5), (6, 8)].iter() {
        assert_eq!(fib(x), y);
    }
}

Answer 4

It's possible to construct tests based on arbitrarily complex parameters and any information known at build time (including anything you can load from a file) with a build script.

We tell Cargo where the build script is:

Cargo.toml

[package]
name = "test"
version = "0.1.0"
build = "build.rs"

In the build script, we generate our test logic and place it in a file using the environment variable OUT_DIR:

build.rs

fn main() {
    let out_dir = std::env::var("OUT_DIR").unwrap();
    let destination = std::path::Path::new(&out_dir).join("test.rs");
    let mut f = std::fs::File::create(&destination).unwrap();

    let params = &["abc", "fooboo"];
    for p in params {
        use std::io::Write;
        write!(
            f,
            "
#[test]
fn {name}() {{
    assert!(true);
}}",
            name = p
        ).unwrap();
    }
}

Finally, we create a file in our tests directory that includes the code of the generated file.

tests/generated_test.rs

include!(concat!(env!("OUT_DIR"), "/test.rs"));

That's it. Let's verify that the tests are run:

$ cargo test
   Compiling test v0.1.0 (...)
    Finished debug [unoptimized + debuginfo] target(s) in 0.26 secs
     Running target/debug/deps/generated_test-ce82d068f4ceb10d

running 2 tests
test abc ... ok
test fooboo ... ok

Answer 5

Without using any additional packages, you can do it like this, since you can write tests that return a Result type

#[cfg(test)]
mod tests {
    fn test_add_case(a: i32, b: i32, expected: i32) -> Result<(), String> {
        let result = a + b;
        if result != expected {
            Err(format!(
                "{} + {} result: {}, expected: {}",
                a, b, result, expected
            ))
        } else {
            Ok(())
        }
    }

    #[test]
    fn test_add() -> Result<(), String> {
        [(2, 2, 4), (1, 4, 5), (1, -1, 0), (4, 2, 0)]
            .iter()
            .try_for_each(|(a, b, expected)| test_add_case(*a, *b, *expected))?;

        Ok(())
    }
}

You will even get a nice error message:

    ---- tests::test_add stdout ----
Error: "4 + 2 result: 6, expected: 0"
thread 'tests::test_add' panicked at 'assertion failed: `(left == right)`
left: `1`,
right: `0`: the test returned a termination value with a non-zero status code (1) which indicates a failure', /rustc/59eed8a2aac0230a8b53e89d4e99d55912ba6b35/library/test/src/lib.rs:194:5

Answer 6

Use https://github.com/frondeus/test-case crate.

Example:

#[test_case("some")]
#[test_case("other")]
fn works_correctly(arg: &str) {
    assert!(arg.len() > 0)
}

Answer 7

Riffing off that great answer by Chris Morgan above, I offer my use of it here. Apart from minor refactoring, this extension allows for an evaluator function which gathers the "actual" value from the system under test and compares it against expectation. The output is pretty nice. My VS Code setup automatically expands the macro invocation into a list of tests functions that may be individually invoked within the editor. In any event, since label becomes the corresponding test function name, cargo test does allows easy test selection as in, cargo test length.

macro_rules! test_cases {
    ( $($label:ident : $eval:ident $exp:expr, $inp:expr);* $(;)? ) => {
        $(
            #[test]
            fn $label() {
                assert!($eval($exp, $inp));
            }
        )*
    }
}

//************************
// test case and test runner for check_len
//************************

test_cases! {
    length__1: check_len     0    , ""; //comments are permitted
    length__2: check_len     14   , "not 14 long";
}
fn check_len(n:usize, s: &str) -> bool {
    n == s.len()
}

//************************
// test case and tester runner for check_content
//************************

test_cases! {    
    content_1: check_content "Bz" , "AB";
    content_2: check_content "567", "123456789";
}
fn check_content(key: &str, s: &str) -> bool {
    s.contains(key)
}

Output...

running 4 tests
test content_1 ... FAILED
test length__1 ... ok
test length__2 ... FAILED
test content_2 ... ok

failures:

---- content_1 stdout ----
thread 'content_1' panicked at 'assertion failed: check_content(\"Bz\", \"AB\")', src/lib.rs:27:1
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

---- length__2 stdout ----
thread 'length__2' panicked at 'assertion failed: check_len(14, \"not 14 long\")', src/lib.rs:22:1


failures:
    content_1
    length__2

test result: FAILED. 2 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s

Answer 8

EDIT: This is now on crates.io as parameterized_test::create!{...} - Add parameterized_test = "0.2.0" to your Cargo.toml file.

---

Building off Chris Morgan’s answer, here's a recursive macro to create parameterized tests (playground):

macro_rules! parameterized_test {
    ($name:ident, $args:pat, $body:tt) => {
        with_dollar_sign! {
            ($d:tt) => {
                macro_rules! $name {
                    ($d($d pname:ident: $d values:expr,)*) => {
                        mod $name {
                            use super::*;
                            $d(
                                #[test]
                                fn $d pname() {
                                    let $args = $d values;
                                    $body
                                }
                            )*
                        }}}}}}}

You can use it like so:

parameterized_test!{ even, n, { assert_eq!(n % 2, 0); } }
even! {
    one: 1,
    two: 2,
}

parameterized_test! defines a new macro (even!) that will create parameterized tests taking one argument (n) and invoking assert_eq!(n % 2, 0);.

even! then works essentially like Chris' fib_tests!, though it groups the tests into a module so they can share a prefix (suggested here). This example results in two tests functions, even::one and even::two.

This same syntax works for multiple parameters:

parameterized_test!{equal, (actual, expected), {
    assert_eq!(actual, expected); 
}}
equal! {
    same: (1, 1),
    different: (2, 3),
}

The with_dollar_sign! macro used above to essentially escape the dollar-signs in the inner macro comes from @durka:

macro_rules! with_dollar_sign {
    ($($body:tt)*) => {
        macro_rules! __with_dollar_sign { $($body)* }
        __with_dollar_sign!($);
    }
}

I've not written many Rust macros before, so feedback and suggestions are very welcome.