Running “cabal test” passes although there are test failures

Question

I have a Cabal test target:

test-suite Tests
  type:              exitcode-stdio-1.0
  main-is:           Main.hs
  hs-source-dirs:    test, src
  build-depends:     base, …
  default-language:  Haskell2010

And a simple testing Main.hs:

import Test.HUnit
testSanity = TestCase $ assertEqual "Should fail" 2 1
main = runTestTT testSanity

Now running cabal test passes:

Test suite Tests: RUNNING...
Test suite Tests: PASS
Test suite logged to: dist/test/Project-0.1.0-Tests.log
1 of 1 test suites (1 of 1 test cases) passed.

Even though there are failures correctly logged in the test suite log:

Test suite Tests: RUNNING...

Cases: 1  Tried: 0  Errors: 0  Failures: 0

### Failure:
Should fail
expected: 2
 but got: 1
Cases: 1  Tried: 1  Errors: 0  Failures: 1
Test suite Tests: PASS
Test suite logged to: dist/test/Project-0.1.0-Tests.log

What am I doing wrong?

NovaDenizen · Accepted Answer · 2015-07-18T16:42:49.160

7

main has to be of type IO a where a can be anything, and that value is not tied in any way to the POSIX exit status of the program. You need to look at the Counts returned by runTest and explicitly choose a successful or failed output code.

import System.Exit

main :: IO ()
main = do
    cs@(Counts _ _ errs fails) <- runTestTT testSanity
    putStrLn (showCounts cs)
    if (errs > 0 || fails > 0) 
        then exitFailure
        else exitSuccess

edited Jul 18 '15 at 16:42

answered Jul 18 '15 at 16:32

NovaDenizen

5,089
14
28

This seems reasonable, but could you add some explanation for future readers? – icktoofay Jul 18 '15 at 16:35
Thanks! I guess it boils down to the `exitcode-stdio-1.0` “driver” specified in the Cabal file. It reports the result of the testing suite according to an exit code. Is there a simpler way to integrate the HUnit tests into Cabal without so much boilerplate, maybe a specialized driver? – zoul Jul 18 '15 at 16:38
The developers of HUnit, like developers of basically every other Haskell library, see process termination as something outside the scope of HUnit. – NovaDenizen Jul 18 '15 at 16:53
@zoul [Tasty](https://hackage.haskell.org/package/tasty)'s [`defaultMain`](https://hackage.haskell.org/package/tasty-0.11.0.3/docs/Test-Tasty.html#v:defaultMain) (with [Tasty-hunit](https://hackage.haskell.org/package/tasty-hunit)) is exactly what you were looking for. – NovaDenizen Jun 28 '16 at 15:01

score 4 · Answer 2 · edited May 23 '17 at 12:31

I believe exitcode-stdio-1.0 expects the test to communicate failure via the program's exit code. However, if you run the test program manually from the shell (look in the dist/build directory) I think you'll find that the exit code is always 0.

Here is a related SO question which suggests that your test should call exitFailure if your suite doesn't pass:

QuickCheck exit status on failures, and cabal integration

FWIW, it works correctly when using stack:

$ mkdir new-dir
$ cd new-dir
$ stack new
(edit new-template.cabal to add HUnit as a test dependency)
(add test case to test/Spec.hs)
$ stack test

Perhaps stack is also scanning the test output.

Running “cabal test” passes although there are test failures

2 Answers2

Linked