How can I silently catch panics in QuickCheck tests?

Question

In the tests of my overflower_support crate, I have found that I get a lot of spurious reports of panics which are already handled using std::panic::catch_unwind(_). This is a bit unfortunate, as it obscures the real errors that may happen. The messages look like:

thread 'safe' panicked at 'arithmetic overflow', src/lib.rs:56

To quell those distracting messages, I introduced the dont_panic(..) function, which hijacks the panic handler, calls a closure and resets the panic handler when done, returning the closures result. It looks like this:

fn dont_panic<F, A, R>(args: A, f: F) -> R
    where F: Fn(A) -> R
{
    let p = panic::take_hook();
    panic::set_hook(Box::new(|_| ()));
    let result = f(args);
    panic::set_hook(p);
    result
}

However, using this function within the function to check somewhat surprisingly not only quells the desired messages, but also quickcheck's error output, which is obviously valuable to me. This occurs even when limiting tests to one thread.

#[test]
fn test_some_panic() {
    fn check(x: usize) -> bool {
        let expected = if x < 256 { Some(x) } else { None };
        let actual = dont_panic(|| panic::catch_unwind(|| { assert!(x < 256); x }).ok());
        expected == actual
    }
    quickcheck(check as fn(usize) -> bool);
}

How can I hide the caught panics from my code while keeping QuickCheck's panics visible?

Could you include a reproducible example of code? If you're wrapping `quickcheck::quickcheck` in `dont_panic`, then it makes sense that your handler is squashing the error messages since the contract of `quickcheck` is to panic on failure. Perhaps you might get more mileage out of `QuickCheck.quicktest`, which returns a `Result` instead of panicing. — BurntSushi5, Jul 21 '16 at 22:05
I'm wrapping the `panic::catch_unwind(_)` call in `dont_panic(..)` within the `fn check((usize, usize)) -> bool` function to be checked by `quickcheck::quickcheck` — llogiq, Jul 21 '16 at 22:11
Added a mostly complete example (should run once you `use std::panic;`). — llogiq, Jul 21 '16 at 22:34

score 2 · Answer 1 · edited Jul 23 '16 at 23:10

2

The default panic handler is printing panic information unconditionally on stderr.

You want to register your own handler.

edited Jul 23 '16 at 23:10

Shepmaster

388,571
95
1,107
1,366

answered Jul 23 '16 at 19:02

dpc.pw

3,462
1
19
24

That's what I do in `dont_panic(_)`. However, this unfortunately appears to hide panic information even after it should have been taken. – llogiq Jul 23 '16 at 23:21
1

Oh. Well, unless you have a way to use eg. a thread local flag, or `PanicInfo`passed to your panic handler to somehow tell one from the other, I don't think there's any other way. – dpc.pw Jul 25 '16 at 05:09

mzabaluev · Answer 2 · 2017-11-29T22:21:38.350

I've met the same problem and a few others, and I ended up writing a crate to solve them:

panic-control

With it, your example might be solved by running in a "quiet" thread (assuming you weren't interested in using catch_unwind specifically):

use panic_control::spawn_quiet;

#[test]
fn test_some_panic() {
    fn check(x: usize) -> bool {
        let expected = if x < 256 { Some(x) } else { None };
        let h = spawn_quiet(|| { assert!(x < 256); x });
        let actual = h.join().ok();
        expected == actual
    }
    quickcheck(check as fn(usize) -> bool);
}

score 1 · Accepted Answer · answered Jul 25 '16 at 14:20

There were two problems with my approach:

The tests run in parallel (and quickcheck appears to add some parallelism of its own, as -j 1 appears ineffective to quell the panic messages).
The message gets written (or otherwise suppressed by set_hook(_)) no matter if there's a catch_unwind(_) or not.

However, dpc.pw's idea to distinguish based on files in the panic handler was spot-on. I changed my approach to call an install_handler() function before calling quickcheck(_), which I reproduce here in full:

use std::panic;
use std::sync::{Once, ONCE_INIT};

static HANDLER : Once = ONCE_INIT;

fn install_handler() {
    HANDLER.call_once(|| {
        let p = panic::take_hook();
        panic::set_hook(Box::new(move|info| {
            if info.location().map_or(false, |l| l.file() != "src/lib.rs" &&
                    !l.file().ends_with("/num/mod.rs")) {
                p(info);
            }
        }));
    })
}

This will quell the panic messages if the panic came from src/lib.rs (which is my overflower_support code) or somewhere from /num/mod.rs (because the Rust libcore code may panic, too).

Note that you could omit the Once, but this would add the handler multiple times and increase the size of stack traces considerably while exacerbating test performance.

How can I silently catch panics in QuickCheck tests?

3 Answers3

Linked