3

We have a project managed in Gitlab, with CI pipeline for builds and tests (pytest, Google tests). Two or three of our test cases in Google tests fail. But Gitlab consider that the test stage is successful. Is it because the success percentage is more than 90% (an arbitrary value) ? Is there a way to make the stage (and thus the complete pipeline) fail if we don't get 100% of success ?

Here is a screenshot of the pipeline summary: enter image description here

Here is the yml script of the stage:

test_unit_test:
  stage: test
  needs: ["build", "build_unit_test"]
  image: $DOCKER_IMAGE
  rules:
    - if: '$CI_PIPELINE_SOURCE != "merge_request_event"'
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
  script: |
    ZIPNAME=`cat _VERSION_.txt`
    ./scripts/gitlab-ci/stage-unittests.sh test_unit_test_report.xml $ZIPNAME
  artifacts:
    reports:
      junit: test_unit_test_report.xml
    expire_in: 1 week

Thank you for any help. Regards.

Adam Marshall
  • 6,369
  • 1
  • 29
  • 45
pamplemousse_mk2
  • 178
  • 1
  • 11

1 Answers1

4

Gitlab CI/CD jobs don't care what the script areas are doing (so they don't look at, for example, test pass percentages). The only they things use to determine if a job passed or failed are exit codes and the allow_failure keyword.

After each command in the before_script, script, and after_script sections are executed, the Gitlab Runner checks the exit code of the command to see if it is 0. If it is non-zero, the command is considered a failure, and if the allow_failure keyword is not set to true for the job, the job fails.

So, for your job, even though the tests are failing, somehow the script is existing with exit code 0, meaning the command itself finished successfully. The command in this case is:

ZIPNAME=$(cat _VERSION_.txt)
./scripts/gitlab-ci/stage-unittests.sh test_unit_test_report.xml $ZIPNAME

NOTE: I replaced your backticks '`' with the $(command) syntax explained here which does the same thing (execute this command) but has some advantages over '`command`' including nesting and easier use in markdown where '`' indicates code formatting.

So, since you are calling a script (./scripts/gitlab-ci/stage-unittests.sh) to run your tests, that script itself is finishing successfully, so the job finishes successfully. Take a look at that script to see if you can tell why it finishes successfully even though the tests fail.

Adam Marshall
  • 6,369
  • 1
  • 29
  • 45
  • 1
    Hello, thank you for the answer. I understand now. I modified my script this morning and now the pipeline fails as expected. Actually, the original `gitlab-ci.yml` contains all the code. Then I put the code in `stage-unittests.sh` and thought that Gitlab just needed to the XML report. To fix the issue, `stage-unittests.sh` returns now the returned value of our Google Test executable. – pamplemousse_mk2 Oct 18 '21 at 12:39
  • 1
    Thank you also for the use of `$(command)`. I didn't know that. This syntax is effectively better than the ticks syntax. – pamplemousse_mk2 Oct 18 '21 at 12:40