0

I'm setting up unit test framework using ctest and cmake. The idea is to have the test command executed in docker container and the test will execute inside the container. That is the requirement.

the add_test looks like this add_test(test_name, /bin/sh, runner.sh test_cmd)

where runner.sh is the script that runs the container and test_cmd is the test command that runs in container.

test_cmd is like this

/path/to/test/test_binary; CODE=$?; echo $CODE > /root/result.txt;

runner.sh has this code

docker exec -t -i --user root $CONTAINERNAME bash -c "test_cmd"

runner.sh further tries to read /root/result.txt from container.

runner.sh spawns new container for each test. Each test runs in its own container So there is no way they can interfere with one another when executed in parallel. /root/result.txt is separate for each container.

when I run the tests like this

make test ARGS="-j8"

for some specific tests /root/result.txt is not generated. Hence the reading fails from that file ( docker exec for test_cmd already returns ) And I cannot see stdout of those tests in LastTest.log

when I run the tests like this

make test ARGS="-j1"

All tests pass. /root/result.txt is generated for all tests and I can see the output (stdout) of those tests

same behavior is there for j > 1. Tests are not being timed out. I checked. My guess is that, before echo $CODE > /root/result.txt; I'm trying to read the exit status from /root/result.txt but again how does it pass in -j1 and in sh its sequential execution. Until one command exits it doesn't move ahead.

One interesting observation is that when I try to do it (docker exec, same command) from python script using subprocess instead of bash it works.

def executeViaSubprocess(cmd, doOutput=False):
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
    stdout, stderr = p.communicate()
    retCode = p.returncode
Suraj
  • 184
  • 1
  • 14
  • `echo $CODE > /root/result.txt` **rewrites** the file for every new test. What is meaning of this? Do you mean `echo $CODE >> /root/result.txt` instead, which *appends* to the file? – Tsyvarev Jul 06 '19 at 08:11
  • yeah. My logic is correct. echo $CODE > /root/result.txt is separate for each test. Each test executes in its own container – Suraj Jul 06 '19 at 08:29
  • Are you sure about "runner.sh spawns new container for each test. Each test runs in its own container". You seem to call all tests in same container, as `$CONTAINERNAME` is constant (and coming from `ctest` environment) between test runs, right?. Container with same name will be reused in constructive calls. You'd have to create multiple containers from the same docker image. https://stackoverflow.com/questions/23735149/what-is-the-difference-between-a-docker-image-and-a-container – R2RT Jul 06 '19 at 14:04
  • PS. You should use `docker run $IMAGENAME` instead, it will create independent container for each call. – R2RT Jul 06 '19 at 14:11
  • @R2RT $CONTAINERNAME is same as the testname. I've verified that multiple containers get spawned. Its actually like this add_test(test_name, /bin/sh, runner.sh test_cmd). so runner.sh gets testname from cli argument and CONTAINERNAME is initialized to testname – Suraj Jul 06 '19 at 15:30
  • You don't show conversion cli argument to environment variable anywhere in your code, so I don't know how "CONTAINERNAME is initialized to testname". Btw, there is no need for `,`s in CMake code. I can test your code on my machine if you provide some MVCE. – R2RT Jul 06 '19 at 15:38
  • @R2RT CONTAINERNAME=$1 thats how it is initialized. note that for each test the script is run. I cannot share whole code with you because of company policies – Suraj Jul 06 '19 at 16:30
  • MVCE is not whole code, but relevant part. Minimal example which implements logic you have already described here. Good thing is you may find out the issue while implementing it. – R2RT Jul 06 '19 at 17:54

0 Answers0