1

I am using Haskell Test Framework through Stack to evaluate QuickCheck properties. When I run stack test, failing properties are reported in the form of Gave up! Passed only 95 tests. The many examples of property testing I've found report failures in the form of Falsifiable, after 48 tests followed by the arguments that failed. These examples, however, seem to be running QuickCheck directly instead of through Stack and HTF.

How can I configure my environment to report the arguments generated by QuickCheck that failed to satisfy the property under test? As pointed out in Testing with HTF, documentation is already sparse and poor for some of these tools alone, let alone for combining them together.

Community
  • 1
  • 1

2 Answers2

2

"Gave up!" means a different kind of failure than "Falsifiable".

QuickCheck has a way of discarding test cases you consider "improper", counting neither towards actual success or failure. A typical source of such discards comes from using the implication operator (==>), where tests cases which do not satisfy the precondition are discarded: "success" is only counted when the precondition is satisfied, to give you a better idea of the extent to which you are testing the postcondition to the right (which is likely the part that actually matters to you as a user). Explicit use of the discard property is also possible, with a different meaning from an actual failure such as returning False.

Discarded tests thus do not falsify the property as a whole (an implication with a false precondition is logically true), but too many discarded tests may result in insufficient coverage, which is signaled through the failure you observed, and there is no counterexample to print. To resolve this failure, find where the discards are coming from, possible outcomes include:

  • use a better generator (avoiding discards);
  • raise the discard threshold, @stefanwehr shows how to do this in HTF in the other answer;
  • these discards should actually be failures.
Li-yao Xia
  • 31,896
  • 2
  • 33
  • 56
  • As to be expected, the real problem was my set of assumptions. Thank you. –  Apr 12 '17 at 00:46
  • I am honestly struggling to find any information on how to specify options to HTF. Is this intended to be passed via command line? Is there a programmatic option as well? I can't find any complete documentation for these tools and it is very frustrating. –  Apr 14 '17 at 14:49
  • Sorry, I confused HTF which you were asking about with test-framework which I ended up talking about. @stefanwehr has the answer about how to set that particular option. There doesn't seem to be a built-in way of specifying it from the command line. – Li-yao Xia Apr 14 '17 at 15:17
1

@Li-yao Xia is right by saying that your generator generates to many discardable test cases. To raise the discard threshold with HTF, you would write your property like this:

prop_somePropertyWithRaisedDiscardThreshold =
    withQCArgs (\args -> args { maxDiscardRatioy = 1000 })
    somePredicateOrProperty

The args variable has type Args, directly from the quickcheck package.

stefanwehr
  • 390
  • 2
  • 7