0

I have a certain amount of time to test a system. Can I write a Python property test that runs property tests until one hour is up? I looked for a solution in hypothesis but I couldn't find one.

I imagine that property-testing libraries have some kind of test-case generator, in which can I could just pull and execute from it until the timeout is up. This would be an acceptable lazy solution.

Matthew Piziak
  • 3,430
  • 4
  • 35
  • 49

1 Answers1

2

Hypothesis does not have a generator you can pull from - the internals are implemented quite differently to the usual lazy-infinite-list construction used by Quickcheck (because... Haskell).

We have an open issue to add a fuzzing mode, which will (hopefully) be the subject of an undergrad group project at Imperial College in early 2019.

Until that's ready, you can add @settings(timeout=60*60, suppress_health_check=HealthCheck.all(), max_examples=10**9) to a test and it should run for an hour (or until finding a bug).

Zac Hatfield-Dodds
  • 2,455
  • 6
  • 19
  • Note that the timout value [has been deprecated](https://hypothesis.readthedocs.io/en/latest/settings.html?highlight=timeout#hypothesis.settings.timeout) so this answer is no longer valid. @Zac Hatfield-Dodds is there an alternative for this? – Ivo Merchiers Mar 26 '19 at 13:15
  • 1
    We removed (it's a noop) the `timeout` setting because it interacted badly with the `max_examples` setting, and we already had a setting for per-test-case `deadline`. So strictly speaking "run for an hour" is no longer a supported use-case, but you can emulate it by tracking time and calling `pytest.skip()` (or `raise unittest.SkipTest` after an hour - they abort without causing a failure. Or wait for https://github.com/HypothesisWorks/hypothesis/issues/171 – Zac Hatfield-Dodds Mar 28 '19 at 00:43