I run "cabal test" with a failing case but it always passes

Question

I want to test a function and I want this test to fail. After initiating the project using cabal init, I created my findOddInt function in the Main.hs file in the app directory:

module Main where

findOddInt :: [Int] -> Int
findOddInt (x:xs) = undefined

main :: IO ()
main = putStrLn "Hello, Haskell!"

I created a directory named tests and created the FindOddInt.hs file:

module FindOddInt ( main ) where
import Main hiding ( main )
import Test.HUnit
import qualified System.Exit as Exit

test1 :: Test
test1 = TestCase (assertEqual "should return an odd integer" 3 (findOddInt [0, 1, 0, 1, 0]))

tests :: Test
tests = TestList [TestLabel "test1" test1]

main :: IO ()
main = do
    result <- runTestTT tests
    if failures result > 0 then Exit.exitFailure else Exit.exitSuccess

my .cabal file is as follows:

cabal-version:      2.4
name:               find-odd-int
version:            0.1.0.0

-- A short (one-line) description of the package.
-- synopsis:

-- A longer description of the package.
-- description:

-- A URL where users can report bugs.
-- bug-reports:

-- The license under which the package is released.
-- license:
author:             André Ferreira
maintainer:         andresouzafe@gmail.com

-- A copyright notice.
-- copyright:
-- category:
extra-source-files: CHANGELOG.md

executable find-odd-int
    main-is:          Main.hs

    -- Modules included in this executable, other than Main.
    -- other-modules:

    -- LANGUAGE extensions used by modules in this package.
    -- other-extensions:
    build-depends:    base ^>=4.14.3.0
    hs-source-dirs:   app
    default-language: Haskell2010

test-suite tests
    type: exitcode-stdio-1.0
    main-is: FindOddIntTest.hs
    build-depends: base ^>=4.14, HUnit ^>=1.6
    hs-source-dirs: app, tests
    other-modules: Main
    default-language: Haskell2010

All set, I ran cabal configure --enable-tests && cabal test and got the output:

Build profile: -w ghc-8.10.7 -O1
In order, the following will be built (use -v for more details):
 - find-odd-int-0.1.0.0 (test:tests) (file app/Main.hs changed)
Preprocessing test suite 'tests' for find-odd-int-0.1.0.0..
Building test suite 'tests' for find-odd-int-0.1.0.0..
[1 of 2] Compiling Main             ( app/Main.hs, /home/asf/Projects/codewars-exercises/haskell/6-kyu/find-odd-int/dist-newstyle/build/x86_64-linux/ghc-8.10.7/find-odd-int-0.1.0.0/t/tests/build/tests/tests-tmp/Main.o )
[2 of 2] Compiling FindOddInt       ( tests/FindOddIntTest.hs, /home/asf/Projects/codewars-exercises/haskell/6-kyu/find-odd-int/dist-newstyle/build/x86_64-linux/ghc-8.10.7/find-odd-int-0.1.0.0/t/tests/build/tests/tests-tmp/FindOddInt.o ) [Main changed]
Linking /home/asf/Projects/codewars-exercises/haskell/6-kyu/find-odd-int/dist-newstyle/build/x86_64-linux/ghc-8.10.7/find-odd-int-0.1.0.0/t/tests/build/tests/tests ...
Running 1 test suites...
Test suite tests: RUNNING...
Test suite tests: PASS
Test suite logged to:
/home/asf/Projects/codewars-exercises/haskell/6-kyu/find-odd-int/dist-newstyle/build/x86_64-linux/ghc-8.10.7/find-odd-int-0.1.0.0/t/tests/test/find-odd-int-0.1.0.0-tests.log
1 of 1 test suites (1 of 1 test cases) passed.

I already looked the following posts:

• Running “cabal test” passes although there are test failures
• Create and run a minimal test suite in Haskell using Hunit only

These posts don't cover my case. Any help is appreciated.

Answer 1

You included the source of your executable as part of your test suite (hs-source-dirs) and this confuses the compiler. When compiling both tests and regular executables, GHC looks for main in a module named Main, and in this case that is app/Main.hs which does nothing, and your test module is compiled but not actually used.

• Don't put app in the hs-source-dirs of the test suite. And more generally, don't include a directory in more than one component (library, executable, test or benchmark suite), unless you know what you're doing. If you need to reuse code, you can put it in a library and have executable and test suite depend on it.

• The file that you put under main-is: in the .cabal file should include module Main where or no module line. The file name can be anything, but to avoid confusing it with a library module, it may be a good idea to use a lowercase name.

---

If the module is not going to be imported by another module (Main, for example), then you are free to use any filename for it.

--- https://downloads.haskell.org/ghc/latest/docs/users_guide/separate_compilation.html

Answer 2

You're inconsistently trying to declare two different modules as Main - for the purpose of finding the main entry point you want FindOddInt.hs to be considered Main, but for the purpose of importing things you want Main.hs to be considered Main.

In your cabal file you have:

test-suite tests
    type: exitcode-stdio-1.0
    main-is: FindOddInt.hs

This says that the Main module should be in a file called FindOddInt.hs (in one of the hs-source-dirs). However in FindOddInt.hs itself you have:

module FindOddInt ( main ) where
import Main hiding ( main )

So there you're saying that that same file is not the Main module after all, it's module FindOddInt. And to make matters worse you're actually importing from another module called Main (which the compiler is going to have to find by the usual module to filename convention, so it'll look for a Main.hs file in any of the hs-source-dirs).

Apparently the way this ends up being compiled, the compiler ends up actually using the Main.hs file as the definition of the Main module, effectively overriding the main-is: FindOddInt.hs configuration in the cabal file. That means your test suite just runs putStrLn "Hello, Haskell!" which of course succeeds. You can confirm this by running cabal test --test-show-details=direct, so that it shows the output from successful tests as well as failures. I get something like this:

Build profile: -w ghc-9.0.2 -O1
In order, the following will be built (use -v for more details):
 - find-odd-int-0.1.0.0 (test:tests) (first run)
Preprocessing test suite 'tests' for find-odd-int-0.1.0.0..
Building test suite 'tests' for find-odd-int-0.1.0.0..
Running 1 test suites...
Test suite tests: RUNNING...
Hello, Haskell!
Test suite tests: PASS
Test suite logged to:
/tmp/scratch/dist-newstyle/build/x86_64-linux/ghc-9.0.2/find-odd-int-0.1.0.0/t/tests/test/find-odd-int-0.1.0.0-tests.log
1 of 1 test suites (1 of 1 test cases) passed.

You can see the "Hello, Haskell" being printed there.

If you change your FindOddInt.hs file so that its header is module Main (since your cabal file says that is the Main module), then you get this error:

<no location info>: error:
    module ‘main:Main’ is defined in multiple files: tests/FindOddInt.hs
                                                     tests/FindOddInt.hs

That's because your other-modules: Main is telling it to look for the Main module again. If you remove that, you get:

Building test suite 'tests' for find-odd-int-0.1.0.0..
Module imports form a cycle:
  module ‘Main’ (tests/FindOddInt.hs) imports itself

This is because the import Main is now being resolved against FindOddInt.hs, not Main.hs.

Basically, there is no way to make this work the way you seem to want it to. It is arguably a bug in Cabal and/or GHC that your original config compiles at all; it should possibly report an error about the mismatch between your cabal files's main-is and the referenced file's own decalared module name (and if not, it's extremely surprising that it uses another file as Main instead of that one). But no amount of fixing problems upstream is going to solve the fundamental contradiction in what you're trying to do. Either FindOddInt.hs is Main (in which case you can't import things from another Main), or it isn't.

In my experience, it is not really a good idea to ever import something from Main. A source file for Main can only really be included in a single application, so as soon as you want 2 application entry points (like your normal executable and a test suite), any code you want to share between them can't be in Main.

What this ends up meaning for my coding practices is that I only put very minimal code in Main that I don't intend to ever test with a library-level test suite (I might do further end-to-end testing of the whole binary, but that's different). Most of my code is typically in an internal library, so it can be imported by both my Main module and by my test suite; my actual app folder will usually only have a very thin Main.hs that stitches together a definition for main that is just combining imported things from the library (sometimes even it's just main = SomeModule.realMain, if I wanted run library tests on realMain).

I also agree with Li-yao Xia's recommendation that you don't put the same folder in the hs-source-dirs of multiple components. It's possible to do that sensibly (and is one option for giving your test suite access to modules that a library doesn't expose), but it also has drawbacks even when it doesn't cause errors like this one (mainly that you waste time compiling files multiple times for each component, instead of compiling once and then linking them). One folder per component is the simplest and easiest way to structure your project, unless you have a good reason otherwise.

For myself, I don't like giving Main a different name than Main.hs. You still can't have another Main.hs that is a different module, so I think it avoids confusion all round to stick firmly to the usual module naming convention and have the Main.hs filename be occupied by the Main module. You likely would have spotted what was happening if you had done that and tried to have a Main.hs importing from Main (and the compiler certainly would have spotted the two files claiming to be Main, as in an error I quoted above). There's not much additional safety in putting Main in a file with a name that can't be imported; it just enforces the "don't import from Main" rule that you end up having to stick to anyway. This makes main-is: Main.hs a pointless bit of boilerplate, but at least you don't have to think about it.