1

I'm working on a TCL project, for which version control is based on Git. So, in order to produce good-quality code, I set up put execution of the tests in pre-commit hook.

However, even if they are executed (trace is shown in command-line), and one of the tests is failed, Git performs the commit. So I launched the hook manually to check the error code, and I figured out that it is null, explaining why Git does not stop:

$ .git/hooks/pre-commit
++++ FlattenResult-test PASSED
(...)
==== CheckF69F70 FAILED
==== Content of test case:
(...)
==== CheckF69F70 FAILED

$ echo $?
0

(Launching the tests script with tclsh also results in $? to be 0.)

So my question is about this last line: why is $? equal to 0, when one of the tcl tests is failed? And how can I achieve a simple pre-commit hook that stops on failure?

I read and reread the tcltest documentation, but saw no setting or information about this error code. And I would really like not to have to parse the tcl tests output, to check if ERROR or FAILED is present...

Edit: versions

  • TCL version : 8.5
  • tcltest version: 2.3.4
Joël
  • 2,723
  • 18
  • 36

3 Answers3

3

This depends on how you run your test suite. Normally you run a file called tests/all.tcl which may look something like this:

package require Tcl 8.6
package require tcltest 2.5

namespace import tcltest::*

configure -testdir [file dirname [file normalize [info script]]] {*}$argv
runAllTests

That final runAllTests returns a boolean indicating success (0) or failure (1). You can use that to generate an exit code by changing the last line to:

exit [runAllTests]
Schelte Bron
  • 4,113
  • 1
  • 7
  • 13
  • 1
    Hmm, it isn't documented that `runAllTests` has a useful return value, but it absolutely should be. That's a (doc) bug… – Donal Fellows Nov 02 '21 at 12:40
  • Thanks for the quick feedback! That gave me hints about how far I was from understanding the documentation. However, as my customer's version of TCL is 8.5 (#sadness) with `tcltest` version 2.3.4, where `runAllTests` returns nothing; meaning I cannot directly apply your suggestion :-( – Joël Nov 02 '21 at 13:11
  • A [relevant blog post proivdes a solution for such old `tcltest` version](https://wuhrr.wordpress.com/2013/09/13/tcltest-part-9-provides-exit-code), by using a `tcltest` hook; I'll give it a try. – Joël Nov 02 '21 at 13:14
  • 1
    The tcltest package is plain Tcl and version 2.5.3 only requires Tcl 8.5 or later. So you should be able to just copy the package and use it with the Tcl version of your customer. – Schelte Bron Nov 02 '21 at 16:41
1

I use this redefinition in some of my test scripts:

# Exit non-zero if any tests fail.
# tcltest's `cleanupTests` resets the numTests array, so capture it first.
proc cleanupTests {} {
    set failed [expr {$::tcltest::numTests(Failed) > 0}]
    uplevel 1 ::tcltest::cleanupTests
    if {$failed} then {exit 1}
}
glenn jackman
  • 238,783
  • 38
  • 220
  • 352
  • Nice! Some questions though: -- 1) does this stop at first failed test file? (Which would look like a drawback to me) -- 2) How would you consider it better than using `cleanupTestsHook`? (existance found in a blog post, linked in [previous comment](https://wuhrr.wordpress.com/2013/09/13/tcltest-part-9-provides-exit-code), also [presented here](https://www.mkssoftware.com/docs/man1/tcltest.1t.asp)) – Joël Nov 02 '21 at 13:42
  • 1) No early exit. – glenn jackman Nov 02 '21 at 16:05
  • 2) Now that I see the Hook, I like it. However, it really relies on you using an `all.tcl`. My suggestion to rewrite the cleanup proc is useful if you run the test files as standalone scripts (where you can actually exit non-zero) -- you wouldn't want to `exit` in the Hook, then you'd bypass all the actual cleanup code. – glenn jackman Nov 02 '21 at 16:06
  • FYI, implementation of the cleanupTestsHook in tcltest: https://github.com/tcltk/tcl/blob/main/library/tcltest/tcltest.tcl#L2425 – glenn jackman Nov 02 '21 at 16:08
1

After some research, I could make it work, even though several factors were against me:

  • I have to use an old TCL version (8.5) with tcltest version 2.3.4, in which runAllTests returns nothing;
  • I forgot to write cleanupTests at the end of test scripts, as the documentation is not really clear about its usage. (It is not clearer now. I just figured out it is needed if you want to get your tests run by runAllTests, which is really not obvious).

And here is my solution, mostly based on Hai's DevBits blog post:

all.tcl

package require tcltest

::tcltest::configure (...)

proc ::tcltest::cleanupTestsHook {} {
  variable numTests
  set ::exitCode [expr {$numTests(Total) == 0 || $numTests(Failed) > 0}]
}

::tcltest::runAllTests
exit $exitCode

Some thoughts about it:

  • I added $numTests(Total) == 0 as a failure condition: this means that no tests was found, which is clearly an erroneous condition;
  • This doesn't catch exceptions in the configuration of the tests, for instance a source command that points to a non-existing file, revealing some failure in tests scaffolding. This would be catched as error in other test framewords (ah, pytest, I miss you!)
Joël
  • 2,723
  • 18
  • 36