0

I am using the hypothesis python package for stateful testing. I am getting the following error when I run my tests:

hypothesis.errors.Flaky: Unreliable assumption: An example which satisfied assumptions on the first run now fails it.

I understand what flaky error means from a similar post. I have a test which failed the first time but passed during the second time. I can understand from the log, which test has led to this failure. Hypothesis tries the same test sequence 4 times during the overall test run among which, 2 of them pass and 2 of them fail.

I have tried the failing test individually without hypothesis and it does not fail. I am trying to understand what leads to the flaky error. Is it possibly a bug in Hypothesis as given in the post below:

What does Flaky: Hypothesis test produces unreliable results mean?

How do I get around this? Please find the log file of the test run at the link: https://github.com/aparnasbose/hypothesis/blob/master/flaky%20test

Aniket
  • 11
  • 4

1 Answers1

2

The problem is almost certainly that your test is not deterministic for all inputs; there are some arguments or sequences of actions that Hypothesis can find which sometimes pass and sometimes fail. Hypothesis considers this a bug in your test, and raises the Flaky error.

To diagnose this in more detail I'd need to see your actual source code.

FYI verbose verbosity is much more useful here than debug (which dumps too much internal state). You may also want upgrade to Hypothesis >= 4.41.1 for improved statistics.

Zac Hatfield-Dodds
  • 2,455
  • 6
  • 19