27

I am using docker compose to run couple of service which depends on each other. Here is part of the docker-compose:

  backend:
    build: .
    command: bash -c "npm run build && npm start"
    ports: 
      - "3015:3015"
    depends_on:
      - couchdb
      - redis
      - uds-mock-server
    volumes:
      - /app/node_modules
      - .:/app
    user: root
  api-test:
    restart: always 
    build: .
    depends_on:
      - backend
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3015/readiness"]
      interval: 200s
      timeout: 200s
      retries: 5
    user: root

As you see I have two service over there and backend should first run and the server needs to be ready then api-test can start. backend has an endpoint: localhost:2015/readiness and whenever it returns 200 then api test can start. When I run while building the order is respected so backend first followed by api-mock but when docker compose starts running them api-test run quicker and since it relies on the backend to be ready it fails.

Base on the following:

Docker Compose wait for container X before starting Y

and

Docker healthcheck in composer file

It is suggested that I should use healthcheck which I do in api test:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:3015/readiness"]
  interval: 200s
  timeout: 200s
  retries: 5

If I get this correctly api-test should run and call the readiness endpoint and wait until it hears from backend readiness (up to 200s) if it fails it waits for 200s and then try again for 5 times. But what I see is api-test keep failing and restarting and does not even give a chance to backend to run and it keeps doing this like a loop. Am I missing anything? Any help is really appreciated

Learner
  • 1,686
  • 4
  • 19
  • 38

3 Answers3

21

The two examples are based on the condition form of depends_on which is no longer supported in compose version 3. So, unless your docker-compose version is <3 the healthcheck will not help you much. The healthcheck sets the status of the container (starting, healthy or unhealthy) but docker-compose does not wait until backend container is healthy before starting the app-test. There is a detailed explanation about how depends_on works in Control startup and shutdown order in Compose

As a side note, the healthcheck in your compose file sets the status of the app-test container and not backend.

Therefore to control when the api-test can start, you have to wrap the service command of the container. For your particular case the following will do the job:

bash -c 'while [[ "$(curl --connect-timeout 2 -s -o /dev/null -w ''%{http_code}'' https://backend:3015/readiness)" != "200" ]]; do echo ..; sleep 5; done; echo backend is up; <service_command>'

It tries to connect to backend every 5 seconds (the connection timeout is 2s). When the received HTTP status code is 200 OK the loop ends and it executes the <service_command>

The relevant docker-compose part:

  api-test:
    restart: always
    command: bash -c 'while [[ "$$(curl --connect-timeout 2 -s -o /dev/null -w ''%{http_code}'' uds-mock-server:4000/readiness)" != "200" ]]; do echo ..; sleep 5; done; echo backend is up;npm start'
    depends_on:
      - backend
      ...

Hope this helps.

b0gusb
  • 4,283
  • 2
  • 14
  • 33
  • 1
    when I run your command I get this: ERROR: Invalid interpolation format for "command" option in service "de-backend": "bash -c 'while [[ "$(curl --connect-timeout 2 -s -o /dev/null -w ''%{http_code}'' https://uds-mock-server:4000/readiness)" != "200" ]]; do echo ..; sleep 5; done; echo backend is up; npm start'" – Learner Dec 03 '19 at 13:42
  • 2
    @Learner In the docker-compose.yml file you should escape the `$` with another `$`. See [Variable Substitution](https://docs.docker.com/compose/compose-file/#variable-substitution). I have updated my answer. – b0gusb Dec 03 '19 at 14:11
  • Thanks. That worked but when I run the compose I see the api-test service keeps sending .. though readiness is readiness is ready with 200 it seems to be an infinite loop so this never happens echo backend is up; npm start – Learner Dec 03 '19 at 14:15
  • What do you get if you run `curl -v uds-mock-server:4000/readiness` inside the api-test container? Is the uds-mock-server the service you wish to check is ready? – b0gusb Dec 03 '19 at 14:22
  • Yes it is mock-server. I changed the code a bit because of requirements changes – Learner Dec 03 '19 at 14:28
  • when I echo the condition: "$(curl --connect-timeout 2 -s -o /dev/null -w ''%{http_code}'' https://backend:3015/readiness)" I get this 000 – Learner Dec 03 '19 at 14:28
  • Try to run curl in verbose mode to see what's going on. – b0gusb Dec 03 '19 at 14:45
  • getting this backend_1 | * Trying xxxx... backend_1 | * TCP_NODELAY set backend_1 | * connect to xxx port 4000 failed: Connection refused backend_1 | * Failed to connect to mock-server port 4000: Connection refused backend_1 | * Closing connection 0 backend_1 | .. [000] – Learner Dec 03 '19 at 14:58
  • ooops my bad http not https Thanks – Learner Dec 03 '19 at 15:26
  • docker supports clean health checks. No need to hack something together. – The Fool Nov 24 '20 at 00:26
19

it's not localhost connection string.

It should be service name of backend container : 

test: ["CMD", "curl", "-f", "http://backend:3015/readiness"]
Thanh Nguyen Van
  • 10,292
  • 6
  • 35
  • 53
  • 1
    This one fixed it for me! – legel Jun 30 '21 at 19:37
  • 1
    is there any way to use semicolon in statement? for example: test: ["CMD", "netstat", "-an | grep 45800 > /dev/null; if [ 0 != $? ]; then exit 1; fi;"] It gives an error to me: ERROR: Invalid interpolation format for "healthcheck" option in service "frontend": "-an | grep 45800 > /dev/null; if [ 0 != $? ]; then exit 1; fi;" – Jaswinder Aug 21 '21 at 04:58
  • This solved it for the, but what does the "-f" flag do? – Ray Jan 29 '22 at 21:53
  • 1
    @Beyar the "-f" flag ignores any error HTTP response that may be returned from the server. Instead it'll just return error 22 code (you can read more about it in the man page) – Martin Jul 24 '22 at 07:02
  • 1
    `healthcheck` should not test foreign container. It is designed to check current container. – Eugen Konkov Oct 22 '22 at 10:20
5

If you are using the latest Docker Desktop (tested on Windows with version 4.14.0) then use depends_on with condition: service_healthy. Example:

    depends_on:
      oracle:
        condition: service_healthy

Note: the version of compose is confusing. Over the years docker-compose became docker compose and alternate implementations were introduced with overlapping version numbers. The version that ships with Docker Desktop now appears to be called docker compose V2, which you indicate in your docker-compose.yml file by NOT including any version number. This differs from the legacy docker-compose V2, which actually defaults to docker-compose V1 if no version is specified in the YAML file.

Ryan
  • 7,499
  • 9
  • 52
  • 61