Why does my HUnit test suite fail but pass successfully in Cabal?

Question

If I have test/Test.hs with

module Main where

import Test.HUnit

test1 :: Test
test1 = TestCase $ assertEqual "Should be one" 1 5

test2 :: Test
test2 = TestCase $ assertEqual "Shold both be zero" 0 0

main :: IO Counts
main = runTestTT $ TestList [test1, test2, test1]

and a .cabal with

test-suite my-test
    type:               exitcode-stdio-1.0
    hs-source-dirs:     test
    main-is:            Test.hs
    build-depends:      base >= 4.8.1.0 && <4.9,
                        HUnit >= 1.3
    default-language:   Haskell2010

and I run cabal test --show-details='always' then I get

Test suite my-test: RUNNING...
### Failure in: 0
test/Test.hs:6
Should be one
expected: 1
 but got: 5
### Failure in: 2
test/Test.hs:6
Should be one
expected: 1
 but got: 5
Cases: 3  Tried: 3  Errors: 0  Failures: 2
Test suite my-test: PASS

Why does my test suite pass when I've had failures? Likewise, if I cabal sdist I get no warning that my tests have failed.

score 6 · Accepted Answer · edited Jan 06 '20 at 21:10

6

According to the Cabal users' guide,

Test suites using the exitcode-stdio-1.0 interface are executables that indicate test failure with a non-zero exit code when run; they may provide human-readable log information through the standard output and error channels.

You've defined

main :: IO Counts
main = runTestTT $ TestList [test1, test2, test1]

This runs tests, prints out test information, and then always exits successfully. If you want Cabal to know the test has failed, you need to capture the Counts, check for errors and failures, and exit with a non-zero status if you find such.

import System.Exit

main :: IO ()
main = do
  results <- runTestTT $ TestList [test1, test2, test1]
  if (errors results + failures results == 0)
    then
      exitWith ExitSuccess
    else
      exitWith (ExitFailure 1)

The test-framework package provides convenient defaultMain functions that do this sort of thing; you may want to consider that approach.

You should note that the exitcode-stdio-1.0 interface is considered semi-deprecated; the Cabal maintainers recommend switching to their rather more Haskellian detailed-0.9 interface.

edited Jan 06 '20 at 21:10

Chris Stryczynski

30,145
48
175
286

answered Oct 02 '15 at 19:01

dfeuer

48,079
5
63
167

I had looked into `detailed-0.9`, but [got the vibe](http://stackoverflow.com/a/18686329/656912) that it was less stable and hard to use. – orome Oct 02 '15 at 19:17
@raxacoricofallapatorius, could be. I've never messed with either; I've only done a bit of expansion of existing test suites in existing projects. – dfeuer Oct 02 '15 at 19:18
@dfeuer: I used your approach with `ExitSuccess` and `ExitFailure`, but it does not work for some reason if I run `cabal test --show-details='always'` - build says test suite : PASS even though there is one failure. Could you please recommend a workaround? – altern Aug 18 '17 at 19:43
@altern, I suggest you ask a separate question with a full reproduction and link to this question. It's too hard to go into detail with comments, and it's impossible to diagnose the problem without code. – dfeuer Aug 19 '17 at 04:14

Why does my HUnit test suite fail but pass successfully in Cabal?

1 Answers1

Linked