How to run test cases for a binary in Makefile

Question

There is a small project which produces a binary application. The source code is C, I'm using autotools to create the Makefile and build the binary - it works as well.

I would like to run tests cases with that binary. Here is what I did:

SUBDIRS = src
dist_doc_DATA = README

TESTS=
TESTS+=tests/config1.conf
TESTS+=tests/config2.conf
TESTS+=tests/config3.conf
TESTS+=tests/config4.conf
TESTS+=tests/config5.conf
TESTS+=tests/config6.conf
TESTS+=tests/config7.conf
TESTS+=tests/config8.conf
TESTS+=tests/config9.conf
TESTS+=tests/config10.conf
TESTS+=tests/config11.conf

I would like to run these cases as argument with the tool. When I run make check, I got:

make[3]: Entering directory '/home/airween/src/mytool'
FAIL: tests/config1.conf
FAIL: tests/config2.conf
FAIL: tests/config3.conf

which is correct, because those files are simple configurations files.

How can I solve that make check runs my tool with the scripts above, and finally I get a list with number of success, failed, ... tests, like in that case:

============================================================================
Testsuite summary for mytool 0.1
============================================================================
# TOTAL: 11
# PASS:  0
# SKIP:  0
# XFAIL: 0
# FAIL:  11
# XPASS: 0
# ERROR: 0

Edit: so I would like to emulate these runs:

for f in `ls -1 tests/*.conf; do src/mytool ${f}; done

but - of course - I want to see the summary at the end.

Thanks.

Please edit your question and show the make rule you have now. Without knowing how you want to use those configuration files, we can't help you: show how you would run the tests from the command line and get that output, then we can help you build a makefile that will do it. — MadScientist, Apr 04 '22 at 21:58
You still didn't show the important parts of the makefile, especially the rule(s). We can't help you by guessing, please edit your question again. — Vroomfondel, Apr 05 '22 at 08:09
@Vroomfondel - I put the full content of Makefile.am in root dir. That's it. — airween, Apr 05 '22 at 08:27
As an aside, that's a [useless use of `ls`](https://wwwiki.fi/era/unix/award.html#ls); you mean simply `for f in tests/*.conf; do src/mytool "${f}" || rc=$$?; done; exit $$rc`. Is there a reason you are not doing that? (Notice how dollar signs need to be doubled in a `Makefile`.) — tripleee, Apr 05 '22 at 08:51
@tripleee - thanks for the tip, but no, this is not what I want. btw this worked for me, but shows the output of each tests, and I don't see how many tests were success or failed, in the summary. — airween, Apr 05 '22 at 10:26
Your question still completely lacks any information about how that summary would be produced. We can't guess from the information you have provided what else to change. — tripleee, Apr 05 '22 at 10:29
I'm not familiar with Automake but https://stackoverflow.com/questions/20440322/how-to-make-make-check-process-tap-output seems to suggest that you should have a tool called something like `test_runner`, can you find that part in the generated `Makefile` and clarify what it currently does? — tripleee, Apr 05 '22 at 10:33
@tripleee I'm not sure that's the solution for my issue. The given example (using `test_runner`) only shows how can you run a unique test case, namely `test_runner`. `make check` will call this tool, you can write your own test/test cases. But I want to run my test program with different configuration files. — airween, Apr 05 '22 at 18:24
And that's the part we are missing. What's the name of your test program and what arguments does it accept? What does `make` actually do when you `make check`? — tripleee, Apr 05 '22 at 18:30
@tripleee the name of my test program is the built binary itself. The arguments are simple text files, configurations... The output of `make check` is in my original post: all test will fail, because `make check` can't run the configuration files. — airween, Apr 05 '22 at 18:32
There is no actual command. That's my question, how can I set up it. I already tested it with `TEST=src/mytool`, which ran successfully, but that's not what I want. — airween, Apr 05 '22 at 18:43
make has no facility for collecting results and showing them in an output format such as you provide. If you want that, you'll have to write that yourself, which has nothing to do with make or makefiles. Once you've written such a command, you can invoke it in your makefile just as you would from the command line (with `$` escaped of course). If your real question is how to write such a script, then you have created a confusing question by referring to make and makefiles: maybe rewrite your question to be more clear about what you really want and remove all the makefile references. — MadScientist, Apr 05 '22 at 18:56
@MadScientist "make has no facility for collecting results and showing them in an output format such as you provide." - it's interesting, because I just put the lines above (`TEST=...`), and `make check` generated the output format. But thanks for your help. — airween, Apr 05 '22 at 19:02
All that means is that your makefile has a target `check`, that someone wrote, that does that collecting for you. That's not part of make: there's no `check` target built into make. That target is supplied by whomever created your makefile. If it already does what you want, then I don't understand what the question is. If it doesn't do what you want, you'll have to change it to make it do what you want. Since you don't show us what the `check` target does, there's little else we can say about it. — MadScientist, Apr 05 '22 at 19:06

score 1 · Accepted Answer · answered Apr 05 '22 at 21:14

The Autotools' built-in test runner expects you to specify the names of executable tests via the make variable TESTS. You cannot just put random filenames in there and expect make or Automake to know what to do with them.

The tests can be built programs, generated scripts, static scripts distributed with the project, or any combination of the above.

How can I solve that make check runs my tool with the scripts above, and finally I get a [test summary report]?

You have acknowledged that your configuration files are not scripts, so stop calling them that! This is in fact the crux of the problem. The easiest solution is probably to create actual executable scripts, one for each case, and name those in your TESTS variable. Each one would run the binary under test with the appropriate configuration file (that is, you're responsible for making them do that if those are the tests you want to perform).

See also the Automake Manual's chapter on tests.

score 0 · Answer 2 · answered Apr 05 '22 at 21:56

Okay, the solution from here:

tests/Makefile.am:
==================

TEST_EXTENSIONS = .conf
CONF_LOG_COMPILER = ./test-suit.sh


TESTS=
TESTS+=config1.conf
TESTS+=config2.conf
TESTS+=config3.conf
TESTS+=config4.conf
TESTS+=config5.conf
TESTS+=config6.conf
TESTS+=config7.conf
TESTS+=config8.conf
TESTS+=config9.conf
TESTS+=config10.conf
TESTS+=config11.conf

test/test-suit.sh:
==================

#!/bin/sh

CONF=$1

exec ../src/mytool $CONF

And the result:

make check
...
PASS: config1.conf
PASS: config2.conf
PASS: config3.conf
PASS: config4.conf
PASS: config5.conf
PASS: config6.conf
PASS: config7.conf
PASS: config8.conf
PASS: config9.conf
PASS: config10.conf
PASS: config11.conf
============================================================================
Testsuite summary for mytool 0.1
============================================================================
# TOTAL: 11
# PASS:  11
# SKIP:  0
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0
============================================================================

This is what I expected.

The [quoting](https://stackoverflow.com/questions/10067266/when-to-wrap-quotes-around-a-shell-variable) in your wrapper script is broken. You don't really need a wrapper for this at all, anyway. `CONF_LOG_COMPILER = ../src/mytool` should suffice. — tripleee, Apr 06 '22 at 05:02
@tripleee thanks for the tip for the wrapper script, you're right. Also thanks for point to the importance of accepting, I did it. Thanks for all to any participant. — airween, Apr 06 '22 at 08:09

How to run test cases for a binary in Makefile

2 Answers2