Test.Framework 'plusTestOptions' and 'testProperty'

Question

I've got an individual QuickCheck2 property I'd like to run more than the standard 100 times, but I've gotten stuck on how to convince Test.Framework to run more using plusTestOptions -- the testProperty fails with "Arguments exhausted after 0 tests".

Relevant code snippet:

mkMsysTests :: TestArgs -> [Test]
mkMsysTests opts =
  [ testGroup "foo"
    [ plusTestOptions testOptions_more (testProperty "bar" prop_bar) ]
  ]

testOptions_more =
  TestOptions { topt_seed = Nothing
              , topt_maximum_unsuitable_generated_tests = Nothing
              , topt_maximum_test_size = Just 500
              , topt_maximum_test_depth = Nothing
              , topt_timeout = Nothing
              , topt_maximum_generated_tests = Just 10000
              }

Theoretically, this should test the property 10,000 times. But it doesn't. Is there any good documentation or example that demonstrates how to use TestOptions to run property tests more than the standard 100 times?

score 2 · Accepted Answer · answered Dec 16 '16 at 19:05

The answer is that one has to increase the number of unsuitable tests, topt_maximum_unsuitable_generated_tests, so that it is greater or equal to the number of generated tests, topt_maximum_generated_tests. Also, should use the mempty instance and modify it with the updated record members. The modified code fragment is:

import           Data.Monoid (mempty)
import           Test.Framework (Test, defaultMainWithArgs, plusTestOptions, testGroup)
import           Test.Framework.Options               (TestOptions' (..))
import           Test.Framework.Providers.HUnit       (testCase)
import           Test.Framework.Providers.QuickCheck2 (testProperty)
import           Test.HUnit                           (Assertion, assertBool, assertFailure)
import           Test.QuickCheck                      (Property, choose, forAll)

{- list of tests -}
mkMsysTests :: TestArgs -> [Test]
mkMsysTests opts =
  [ testGroup "foo"
    [ plusTestOptions testOptions_more (testProperty "bar" prop_bar) ]
  ]

testOptions_more =
  mempty { topt_maximum_unsuitable_generated_tests = Just 10000
         , topt_maximum_generated_tests = Just 10000
         }

I don't have a good explanation why the Test.Framework code gives up if the number of unsuitable tests is less than the number of generated tests; I just did a quick "eyeball" scan on the code and noticed the fix.

Test.Framework 'plusTestOptions' and 'testProperty'

1 Answers1

Linked