4

I have two Docker containers:

  1. b-db - contains my database
  2. b-combined - contains my web application and tests that run once the container is up and running.

I'm using a docker-compose.yml file to start both containers.

version: '3'
services:
    db:
        build:
            context: .
            dockerfile: ./docker/db/Dockerfile
        container_name: b-db
        restart: unless-stopped
        volumes:     
            - dbdata:/data/db
        ports:
            - "27017:27017"
        networks:
            - app-network

    combined:
        build:
            context: .
            dockerfile: ./docker/combined/Dockerfile
        container_name: b-combined
        restart: unless-stopped
        env_file: .env
        ports:
            - "5000:5000"
            - "8080:8080"
        networks:
            - app-network
        depends_on:
            - db

networks:
    app-network:
        driver: bridge

volumes:
    dbdata:
    node_modules:

I'm using Jenkins to launch my containers and start running tests using the following command. I'm using --exit-code-from as outlined by SO posts from here, here and here.

docker-compose up --build --exit-code-from combined

Below is what my Jenkinsfile looks like.

pipeline {
    agent any
    environment {
        CI = 'true'
    }
    stages {
        stage('Test') {
            steps {
                sh 'docker-compose up --build --exit-code-from combined'
            }
        }
    }
}

When my tests run it appears that b-combined exits as expected with a non-zero error code, which is displayed to the console as shown below. This triggers both containers to shutdown, which is also expected behaviour.

b-combined exited with code 2

Stopping b-combined ...

Stopping b-db ...

Stopping b-db ... done Aborting on container exit...

Why is it that Jenkins still displays the tests has having passed (see below screenshot)? Shouldn't Jenkins have failed following a non-zero exit of the docker-compose up --build --exit-code-from combined command?

enter image description here

Furthermore, when I run the following immediately after I run the above docker-compose command in my command line locally (not in Jenkins) I get an error code of 0, which confirms the problem does not lie with Jenkins but rather with docker-compose not recognising that I'm exiting init.sh with a non-zero exit code.

$ echo $?
0

As per the below suggestion from @LinPy, I ran the following command locally on my machine and in Jenkins.

docker-compose up -d --build db && docker-compose up --build combined || exit 2; echo $?

The output I received is as follows. The last line is the output of echo $?, which shows that the script still exits with error code of 0.

b-combined | Mongoose disconnected
b-combined | TEST ENDED WITH EXIT CODE OF: 2
b-combined | EXITING SCRIPT WITH EXIT CODE OF: 2
b-combined exited with code 2
0

Below is a screenshot of Jenkins after the above command is run:

enter image description here

To help with debugging, below is the Dockerfile for the combined service in docker-compose.yml.

RUN npm install

COPY . .

EXPOSE 5000

RUN npm install -g history-server nodemon

RUN npm run build-test

EXPOSE 8080

COPY ./docker/combined/init.sh /scripts/init.sh

RUN ["chmod", "+x", "/scripts/init.sh"]

ENTRYPOINT [ "/scripts/init.sh" ]

Below is what is in my init.sh file.

#!/bin/bash
# Start front end server
history-server dist -p 8080 &
front_pid=$!

# Start back end server that interacts with DB
nodemon -L server &
back_pid=$!

# Run tests
NODE_ENV=test $(npm bin)/cypress run --config video=false --browser chrome

# Error code of the test
test_exit_code=$?

echo "TEST ENDED WITH EXIT CODE OF: $test_exit_code"

# End front and backend server
kill -9 $front_pid
kill -9 $back_pid

# Exit with the error code of the test
echo "EXITING SCRIPT WITH EXIT CODE OF: $test_exit_code"
exit "$test_exit_code"

Below is the Dockerfile for my db service. All its doing is copying some local data into the Docker container and then initialising the database with this data.

FROM  mongo:3.6.14-xenial

COPY ./dump/ /tmp/dump/

COPY mongo_restore.sh /docker-entrypoint-initdb.d/

RUN chmod 777 /docker-entrypoint-initdb.d/mongo_restore.sh

Below is what is in mongo_restore.sh.

#!/bin/bash
# Creates db using copied data
mongorestore /tmp/dump

Following the updated solution from @LinPy, I tried the following steps.

Below is what my new combined Dockerfile looks like:

RUN npm install

COPY . .

EXPOSE 5000

RUN npm install -g history-server nodemon

RUN npm run build-test

EXPOSE 8080

COPY ./docker/combined/init.sh /scripts/init.sh

RUN ["chmod", "+x", "/scripts/init.sh"]

ENTRYPOINT [ "/scripts/init.sh" ]

# NEW LINE ADDED HERE
CMD ["sh", "-c",  "exit $(cat /scripts/exit_code)"]

Below is what my new init.sh file looks like.

#!/bin/bash
# Start front end server
history-server dist -p 8080 &
front_pid=$!

# Start back end server that interacts with DB
nodemon -L server &
back_pid=$!

# Run tests
NODE_ENV=test $(npm bin)/cypress run --config video=false --browser chrome

# Error code of the test
test_exit_code=$?

echo "TEST ENDED WITH EXIT CODE OF: $test_exit_code"

# End front and backend server
kill -9 $front_pid
kill -9 $back_pid

# NEW LINES ADDED HERE
echo "$test_exit_code" > /scripts/exit_code
exec "$@"

# Exit with the error code of the test
echo "EXITING SCRIPT WITH EXIT CODE OF: $test_exit_code"
exit "$test_exit_code"

Finally, I ran the following command:

docker-compose up -d --build db && docker-compose up --build combined || exit 2; echo $?

The output is as follows - last line (from output of echo $?) has exit code of 0.

b-combined | TEST ENDED WITH EXIT CODE OF: 2 ===========================
b-combined exited with code 2
0

SOLUTION:

I was using an older version of docker-compose (pre v1.23.0). As you can see in the release notes of docker-compose, there have been several bug fixes around --exit-code-from since v1.23.0.

ptk
  • 6,835
  • 14
  • 45
  • 91
  • Can you isolate the problem to a specific error rather than asking us to debug both Jenkins, docker, docker-compose, and your scripts together? If your jenkins script is just "exit 2" does that fail? If you run docker-compose locally, does that give the expected return code? If you run your commands in a docker container without compose, do they work as expected? – BMitch Nov 12 '19 at 14:48
  • The zero exit code appears regardless if I run the docker-compose command in Jenkins or locally as I've clarified in my question. It seems to me that exiting my `init.sh` script with a non-zero exit code does not result in the docker-compose command exiting with a non-zero exit code. As you may have observed from @LinPy's solution, we've also tried exiting from `combined`'s Dockerfile with a non-zero exit code instead of from `init.sh` but the problem still exists. – ptk Nov 13 '19 at 00:27
  • To answer your sub-questions, if the Jenkins script is just "exit 2" it does fail (hence the problem is not with Jenkins recognising exit codes). If my `init.sh` script only contains "exit 2" and nothing else, the docker-compose command still exits with a 0 exit code. If I only run the `combined` service (in other words remove `db` service from `docker-compose.yml`), I still exit with a 0 exit code. – ptk Nov 13 '19 at 01:39
  • I've been unsuccessful in recreating your issue locally with just docker-compose and a simple yml file. What version of docker-compose are you running, and have you tried updating to a newer version of not on the latest stable? – BMitch Nov 13 '19 at 01:52

2 Answers2

3

As mentioned in the comments, I've been unable to reproduce your issue with a simple compose file. If the following example still gives you exit code 0, then the issue is likely with your install of docker-compose. And if it works, then the issue will be with your container not actually exiting with the proper exit code. You should also run a docker container ls -a to see the exited containers and their exit codes, and docker logs on the stopped containers to verify the output. Here's my working example:

$ cat docker-compose.exit-code.yml
version: '3'

services:
  good:
    image: busybox
    command: /bin/sh -c "exit 0"

  bad:
    image: busybox
    command: /bin/sh -c "exit 42"

$ docker-compose -f docker-compose.exit-code.yml up --exit-code-from bad
Starting test_good_1_69c61ee0bdc6 ... done
Starting test_bad_1_fbe3194c1994  ... done
Attaching to test_bad_1_fbe3194c1994, test_good_1_69c61ee0bdc6
test_bad_1_fbe3194c1994 exited with code 42
Aborting on container exit...

$ echo $?
42

$ docker-compose -f docker-compose.exit-code.yml up --exit-code-from good
Starting test_good_1_69c61ee0bdc6 ... done
Starting test_bad_1_fbe3194c1994  ... done
Attaching to test_good_1_69c61ee0bdc6, test_bad_1_fbe3194c1994
test_good_1_69c61ee0bdc6 exited with code 0
Aborting on container exit...

$ echo $?
0
BMitch
  • 231,797
  • 42
  • 475
  • 450
  • Yep, I believe it may have to do with the version of docker-compose. I've updated my docker-compose and its spitting out error codes but I'm encountering another issue of docker-compose spitting out 137 error codes when tests complete successfully in the meantime. I'll revert back once I get more time to debug what's actually going on... Thanks for all your help! – ptk Nov 14 '19 at 03:41
  • 1
    https://success.docker.com/article/what-causes-a-container-to-exit-with-code-137 – BMitch Nov 14 '19 at 11:51
  • Thanks again for sharing that article with me. It helped a lot with debugging but I still wasn't able to pinpoint why I keep receiving a 137 error code. I've made a separate SO post here about the problem [here](https://stackoverflow.com/questions/59296801/docker-compose-exit-code-is-137-when-there-is-no-oom-exception) so if you have some time, I'd appreciate if you took a look for me :) Thanks again for helping me resolve this issue btw. – ptk Dec 12 '19 at 02:28
1

I think your command should be:

docker-compose up --build --exit-code-from combined combined

in this way you will get the exit code from combined

since the combined service depends on db that will start the db service also .

I think the exit code is always 0 beacuse --exit-code-from implies --abort-on-container-exit and the db will return 0 on exit

--abort-on-container-exit  Stops all containers if any container was
                           stopped. Incompatible with -d. ...
--exit-code-from SERVICE   Return the exit code of the selected service
                           container. Implies --abort-on-container-exit.

update

try to add this to your script replacing the last line:

#!/bin/bash
# Start front end server
history-server dist -p 8080 &
front_pid=$!

# Start back end server that interacts with DB
nodemon -L server &
back_pid=$!

# Run tests
NODE_ENV=test $(npm bin)/cypress run --config video=false --browser chrome

# Error code of the test
test_exit_code=$?

echo "TEST ENDED WITH EXIT CODE OF: $test_exit_code"

# End front and backend server
kill -9 $front_pid
kill -9 $back_pid

# Exit with the error code of the test
echo "EXITING SCRIPT WITH EXIT CODE OF: $test_exit_code"
echo "$test_exit_code" > /scripts/exit_code
exec "$@"

then add an CMD to your Dockerfile:

CMD ["sh", "-c",  "exit $(cat /scripts/exit_code)"]

RUN it

docker-compose up --build --exit-code-from combined combined 
LinPy
  • 16,987
  • 4
  • 43
  • 57
  • Thanks for your answer! Gave it a shot and it didn't seem to do the job unfortunately :( Error code is still 0 and Jenkins still thinks everything is ok. I think you might be right about db messing with the exit code... hmmmm – ptk Nov 12 '19 at 09:33
  • Thanks for the updated solution. I also gave that a shot but it still got a zero exit code. I've actually updated my question detailing the steps I took to try your updated solution and the output I received. I've also added some more details about my Dockerfiles and my initialisation script. Hopefully that might help you discover something that's causing this... I can't seem to spot anything myself since I'm quite new to Docker :( – ptk Nov 12 '19 at 11:53
  • Yep that's correct. Just tried it locally on my machine and I'm still getting a 0 error code. If it helps I'm using Docker version 18.09.1, build 4c52b90. As far as I'm aware that shouldn't be an issue tho – ptk Nov 12 '19 at 12:09
  • Thanks so much for updating your solution! I'm a little unclear on the positioning of the lines of code you have suggested. I've updated my question to show my attempt at incorporating your changes; however, are you able to confirm if I've placed the new lines in the correct places? If not, feel free to show me where I should be adding the code. Also are you able to confirm that you tested on your machine with `docker-compose up -d --build db && docker-compose up --build combined || exit 2`? Thanks so much for being so patient with me. – ptk Nov 12 '19 at 12:49
  • Followed your instructions exactly and I'm still getting a zero exit code :( I also tried a bunch of other commands in addition to the one you suggested but those didn't work either. I'm going to give your solution an upvote because there's so much useful information for others but unfortunately, I'm still stuck with docker-compose command exiting with 0 exit code... So frustrating! It's really late where I'm at right now so won't be able to try anything else for now but if you think of something I will try tomorrow morning.. Thanks for all your help!! – ptk Nov 12 '19 at 13:27
  • you should run this docker-compose up --build --exit-code-from combined combined . with the last update – LinPy Nov 12 '19 at 13:48