How to set or increase the number of tests to pass using the Test.Framework and defaultMainWithOpts?

Question

There is already several ways to specify the number of tests.

E.g.:

However, how to pass the parameters to defaultMainWithOpts?

Furthermore, is it working at all or can I apply maxSuccess somewhere else?

The most examples that I know off have mempty as second parameter for defaultMainWithOpts.

import qualified Test.Framework as TF
import qualified Test.Framework.Providers.HUnit as FHU
import qualified Test.Framework.Providers.QuickCheck2 as QC2 -- supports Test.QuickCheck with 'testProperty'
import qualified Test.HUnit as HU
import qualified Test.QuickCheck as QC

...

main :: IO ()
main = 
    TF.defaultMainWithOpts
        [
            FHU.testCase "..." (... HU.@?= ...), 
            QC2.testProperty
                "..." 
                (\xyz -> 
                    (
                        (...) 
                        QC.==> 
                        (...) ) ), 
            QC2.testProperty
                "..." 
                (\xyz -> 
                    (
                        (...) 
                        QC.==> 
                        (...) ) )
        ]
        mempty

...

The function defaultMainWithOpts has type [Test] -> RunnerOptions -> IO ().

...and RunnerOptions is defined as

type RunnerOptions = RunnerOptions' Maybe

Hä? What is RunnerOptions' and why is there Maybe?

RunnerOptions' is defined as:

data RunnerOptions' f = RunnerOptions {
    ropt_threads :: f Int
    ropt_test_options :: f TestOptions
    ropt_test_patterns :: f [TestPattern]
    ropt_xml_output :: f (Maybe FilePath)
    ropt_xml_nested :: f Bool
    ropt_color_mode :: f ColorMode
    ropt_hide_successes :: f Bool
    ropt_list_only :: f Bool }

What is this supposed to mean?

Long story short: How to pass the number of tests to defaultMainWithOpts?

Jörg Brüggmann · Answer 1 · 2021-10-09T14:21:59.503

If you substitute RunnerOptions' Maybe than you you see it more clearly.

It could have been defined like this:

data RunnerOptions = RunnerOptions {
    ropt_threads :: Maybe Int
    ropt_test_options :: Maybe TestOptions
    ropt_test_patterns :: Maybe [TestPattern]
    ropt_xml_output :: Maybe (Maybe FilePath)
    ropt_xml_nested :: Maybe Bool
    ropt_color_mode :: Maybe ColorMode
    ropt_hide_successes :: Maybe Bool
    ropt_list_only :: Maybe Bool }

Hence, a valid parameter RunnerOptions parameter is for example:

RunnerOptions Nothing Nothing Nothing Nothing Nothing Nothing Nothing Nothing

or

RunnerOptions Nothing (Just ...) Nothing Nothing Nothing Nothing Nothing Nothing

The parameter you are looking for is topt_maximum_generated_tests and this one is behind ropt_test_options :: Maybe TestOptions.

TestOptions is declared the same way.

type TestOptions = TestOptions' Maybe

data TestOptions' f = TestOptions {
    topt_seed :: f Seed                                 -- ^ Seed that should be used to create random numbers for generated tests
    topt_maximum_generated_tests :: f Int               -- ^ Maximum number of tests to generate when using something like QuickCheck
    topt_maximum_unsuitable_generated_tests :: f Int    -- ^ Maximum number of unsuitable tests to consider before giving up when using something like QuickCheck
    topt_maximum_test_size :: f Int                     -- ^ Maximum size of generated tests when using something like QuickCheck
    topt_maximum_test_depth :: f Int                    -- ^ Maximum depth of generated tests when using something like SmallCheck
    topt_timeout :: f (Maybe Int) }                     -- ^ The number of microseconds to run tests for before considering them a failure

Hence, a valid parameter TestOptions record is:

`TestOptions Nothing Nothing Nothing Nothing Nothing Nothing`

or

`TestOptions Nothing (Just 1000) Nothing Nothing Nothing Nothing`

If you insert this into the runners option then it looks like:

RunnerOptions Nothing (Just (TestOptions Nothing (Just 1000) Nothing Nothing Nothing Nothing)) Nothing Nothing Nothing Nothing Nothing Nothing

The complete main function may look like:

main :: IO ()
main = 
    TF.defaultMainWithOpts
        [
            FHU.testCase "..." ((H.hugeProduct (H.ToHuge 2 3 5 7)) HU.@?= 210), 
            ...
            QC2.testProperty
                "..." 
                (\xyz -> 
                    (
                        (...) 
                        QC.==> 
                        (...) ) ), 
            QC2.testProperty
                "..." 
                (\xyz -> 
                    (
                        (...) 
                        QC.==> 
                        (...) ) )
        ]
    (TF.RunnerOptions 
        {
            TF.ropt_threads = Nothing, 
            TF.ropt_test_options = (Just testOptionsA), 
            TF.ropt_test_patterns = Nothing, 
            TF.ropt_xml_output = Nothing, 
            TF.ropt_xml_nested = Nothing, 
            TF.ropt_color_mode = Nothing, 
            TF.ropt_hide_successes = Nothing, 
            TF.ropt_list_only = Nothing
        })

testOptionsA = 
    TF.TestOptions  
        {
            TF.topt_seed = Nothing, 
            TF.topt_maximum_generated_tests = (Just 1000), 
            TF.topt_maximum_unsuitable_generated_tests = Nothing, 
            TF.topt_maximum_test_size = Nothing, 
            TF.topt_maximum_test_depth = Nothing, 
            TF.topt_timeout = Nothing
        }

Then the console output is like this:

Test5> Test suite Test5-test passed
Completed 2 action(s).
...: [OK]
...
...: [Arguments exhausted after 48 tests]
...: [OK, passed 1000 tests]

         Properties  Test Cases  Total
 Passed  2           5           7
 Failed  0           0           0
 Total   2           5           7

How to set or increase the number of tests to pass using the Test.Framework and defaultMainWithOpts?

1 Answers1