Is there a way to make a unit / integration test fail after a given amount of time?

Question

I am implementing a virtual machine in Rust as an exercise. Some integration tests run small programs on that virtual machine and make assertions about the final result.

However, it can happen that a bug leads to the machine getting stuck in an infinite loop, meaning that cargo test never finishes.

Is there a way to make a unit / integration test fail after a given amount of time? Maybe something similar to #[should_panic], but instead, say, #[time_limit(500)]?

Cargo does not offer such functionality. Aborting a thread is not supported on most platforms, and not supported by `std`. Your second-best option may be to `libc::fork` inside the `#[test]`-function, execute the test in the forked subprocess, and watch the subprocess / `libc::kill` it from the main process. Your best option is probably to integrate a timeout in the `VM`, which gets periodically checked. — user2722968, Jul 30 '21 at 20:56
Your question might be answered by the answers of [Aborting evaluation in Rust after a timeout](https://stackoverflow.com/q/59805874/155423); [What is the correct way in Rust to create a timeout for a thread or a function?](https://stackoverflow.com/q/36181719/155423). If not, please **[edit]** your question to explain the differences. Otherwise, we can mark this question as already answered. — Shepmaster, Jul 30 '21 at 21:00
If you are willing to use external crates, there is https://docs.rs/ntest/0.7.3/ntest/attr.timeout.html that seems to cover your use case. — SirDarius, Jul 30 '21 at 21:00
I'm out of my depth here and don't know much about Rust, but in terms of a fast/pragmatic answer, a lot of CI systems support what you want out of the box in a language agnostic way (including gitlab, jenkins, travis, and circle). A bit annoying if it's a small project that you don't want to use CI for, but having a CI pipe is rarely a bad idea. — Matt Messersmith, Jul 30 '21 at 21:06

Todd · Accepted Answer · 2021-07-31T04:23:47.163

The ntest crate with its #[timeout(<ms>)] macro as mentioned in the comments by SirDarius may be a simpler solution than the following.

As an alternative, there's a crate that allows creation of processes that run closures, which has some test related features. Processes, unlike threads, can be killed if they run too long.

The crate procspawn has this closure feature, and some other nice features useful for tests where timeouts are desired. Another one of these features is serialization of the return value from the code invoked in the child process. The test code can get the result back and check it in a very straightforward way.

The code to be tested:

use std::thread;
use std::time::Duration;

fn runs_long() -> i32
{
    thread::sleep(Duration::from_millis(5000));
    42
}

Within the same file, we can add some tests with timeouts:

#[cfg(test)]
mod tests {
    use std::time::Duration;
    use procspawn;
    
    use crate::*;
    
    procspawn::enable_test_support!();
    
    #[test]
    fn runs_long_passes()
    {
        let handle = procspawn::spawn((), |_| runs_long());
        
        match handle.join_timeout(Duration::from_millis(6000)) {
            Ok(result) => assert_eq!(result, 42),
            Err(e)     => panic!("{}", e),
        }
    }
    
    #[test]
    fn runs_long_fails()
    {
        let handle = procspawn::spawn((), |_| runs_long());
        
        match handle.join_timeout(Duration::from_millis(1000)) {
            Ok(result) => assert_eq!(result, 42),
            Err(e)     => panic!("{}", e),
        }
    }
}

And running the tests with cargo test on the command line, we get:

running 3 tests
test tests::procspawn_test_helper ... ok
test tests::runs_long_fails ... FAILED
test tests::runs_long_passes ... ok

failures:

---- tests::runs_long_fails stdout ----
thread 'tests::runs_long_fails' panicked at 'process spawn error: timed out', src/main.rs:50:17
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    tests::runs_long_fails

test result: FAILED. 2 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 6.01s

error: test failed, to rerun pass '--bin procspawn'

To enable the test support feature, include the following in your Cargo.toml file:

[dependencies]
procspawn = { version = "0.10", features = ["test-support"] }

Is there a way to make a unit / integration test fail after a given amount of time?

1 Answers1