Docker compose getting mysql db fully started before flask app starts

Question

I just started to dockerize my app. I've built my Dockerfile and docker-compose.yml and everything seems to work fine except one thing. There are times my flask app will start too quick and throw a connection refused error (because the MySQL db is not fully up). I am using healthcheck to check if the db is up but this seems to not be reliable (I'm even making sure I can see show databases, but mysql apparently initializes more things after the healthcheck passes? not sure what the healthcheck is for then). In my output, I see that the db does get created first but it is still initializing when the flask app starts up. Ideally, when I run docker-compose up I want to be able to see this line first,

db_1_eae741771281 | 2018-11-10T00:50:21.473098Z 0 [Note] mysqld: ready for connections.

and then start my flask app entry point. Currently, it doesn't do this.

Is there a more reliable way to ensure the MySQL is fully up before starting my start.sh?

Dockerfile:

FROM python:3.5-alpine

RUN apk update && apk upgrade

RUN apk add --no-cache curl python build-base openldap-dev python2-dev python3-dev pkgconfig python-dev libffi-dev musl-dev make gcc

RUN pip install --upgrade pip

RUN adduser -D user

WORKDIR /home/user

COPY requirements.txt requirements.txt
RUN python -m venv venv
RUN venv/bin/pip install -r requirements.txt

COPY app app
COPY start.sh ./
RUN chmod +x start.sh

RUN chown -R user:user ./
USER user

EXPOSE 5000
ENTRYPOINT ["./start.sh"]

docker-compose.yml:

version: "2.1"
services:
  db:
    image: mysql:5.7
    ports:
      - "32000:3306"
    environment:
      - MYSQL_DATABASE=mydb
      - MYSQL_USER=user
      - MYSQL_PASSWORD=user123
      - MYSQL_ROOT_PASSWORD=user123
    volumes:
      - ./db:/docker-entrypoint-initdb.d/:ro
    healthcheck:
            test: "mysql --user=user --password=user123 --execute \"SHOW DATABASES;\""
            timeout: 20s
            retries: 20

  app:
    build: ./
    ports:
      - "5000:5000"
    depends_on:
      db:
        condition: service_healthy

start.sh

#!/bin/sh

source venv/bin/activate
# Start Gunicorn processes
echo Starting Gunicorn.

exec gunicorn -b 0.0.0.0:5000 wsgi --chdir my_app --timeout 9999 --workers 3 --access-logfile - --error-logfile - --capture-output --log-level debug

Possible duplicate of [Docker-compose check if mysql connection is ready](https://stackoverflow.com/questions/42567475/docker-compose-check-if-mysql-connection-is-ready) — Yann39, Nov 10 '18 at 17:14
@Yann39 I already saw that post and tried all the solutions recommended in the post, however, none of those works. As you can see in my code I have the `health_check` in place, but still getting the issue. — lion_bash, Nov 11 '18 at 03:23

score 2 · Accepted Answer · answered Nov 11 '18 at 10:33

2

OK I also had problems with health_check...

Maybe not the most optimal, but the most reliable solution is to use a MySQL client (mysqladmin) to ping your MySQL server before starting your application.

1 - Create a wait.sh script (db is your MySQL service name here) :

#!/bin/sh

# Wait until MySQL is ready
while ! mysqladmin ping -h"db" -P"3306" --silent; do
    echo "Waiting for MySQL to be up..."
    sleep 1
done

2 - Get a MySQL client from your app Dockerfile :

# install mysql client, will be used to ping mysql
apt-get -y install mysql-client

3 - In your docker-compose.yml file, just add scripts to your container (I used volumes but you can keep using COPY) and run wait.sh before start.sh :

app:
    build: ./
    ports:
      - "5000:5000"
    depends_on:
      db:
    command: bash -c "/usr/local/bin/wait.sh && /usr/local/bin/start.sh"
    volumes:
      - ./start.sh:/usr/local/bin/start.sh
      - ./wait.sh:/usr/local/bin/wait.sh

This should work.

If you really don't want having to download a MySQL client, try this (again db is your MySQL service name here). It has worked in most of my project but not in all (may depend of the distribution?) :

#!/bin/sh

# Wait until MySQL is ready
while ! exec 6<>/dev/tcp/db/3306; do
    echo "Trying to connect to MySQL at 3306..."
    sleep 5
done

PS : avoid naming your services "app" or "db", you may have problems later if you have other containers with those same service names (even in different networks).

answered Nov 11 '18 at 10:33

Yann39

14,285
11
56
84

Is that different than this solution here?https://stackoverflow.com/a/42757250/10067412 – lion_bash Nov 11 '18 at 11:17
Yes it does not use `healthcheck` but `command` instead which override the default command. Although both solutions use `mysqladmin` to ping server every _interval_ seconds. – Yann39 Nov 11 '18 at 19:07
Okay, and why do we need to install `mysql-client`? Will `mysqladmin ping` not work without it? – lion_bash Nov 12 '18 at 09:30
Yes you need a client for the `mysqladmin` command to be available from your app container. – Yann39 Nov 12 '18 at 12:28
I'm getting permission denied for `wait.sh`. How can I change the permissions for `wait.sh` to be executable? – lion_bash Nov 12 '18 at 20:03
Ah ok you use another user. So just change permissions like you do for the `start.sh` script : `RUN chmod +x wait.sh`. Note : try to chain `RUN` instructions as much as possible to avoid creating to much layers : `RUN chmod +x start.sh && chmod +x wait.sh && chown -R user:user ./` – Yann39 Nov 12 '18 at 20:22
Is there a way to do this in docker-compose ? I actually moved everything to `volumes`, can you use the `RUN` command in docker compose as well? – lion_bash Nov 12 '18 at 20:26
1

Yes just chain your commands in the docker-compose `command` instruction : `command: bash -c "chmod +x /usr/local/bin/wait.sh /usr/local/bin/start.sh && /usr/local/bin/wait.sh && /usr/local/bin/start.sh"`. You can also mix `wait.sh` and `start.sh` in a single script if you prefer. – Yann39 Nov 12 '18 at 20:56
Thank you so much. One little thing i had to do is install `bash` using `RUN apk add --no-cache bash` for it to work since i was using python3.5-alpine, other than that this works like a charm accepted your answer and upvoted! – lion_bash Nov 12 '18 at 22:35
1

Indeed `python3.5-alpine` is based on `alpine:3.8` Linux image which use `sh` (`bash` is not included). So you can just use `sh` if you don't want to install `bash`, then the docker-compose command would be : `command: sh -c "..."`. Also use the `#!/bin/sh` shebang instead of `#!/bin/bash` in your scripts. For the differences see [this answer](https://stackoverflow.com/questions/5725296/difference-between-sh-and-bash). – Yann39 Nov 13 '18 at 07:34

score 1 · Answer 2 · answered Nov 10 '18 at 16:49

1

While using a health check is easier, it totally depends on how reliable the check is.

Another approach would be to rely on projects like wait-for-it or wait-for, in your app container.
Since you are getting a connection refused, these scripts could return only once the connection is possible and your app can start only after.

Also, in case that doesn't work too, you could have a separate script (python in your case) to check until the DB is ready and you can call this script in your start.sh before starting the flask app.

answered Nov 10 '18 at 16:49

Pramodh Valavala

752
3
9

yeah, the `wait-for-it` script didn't work either. I ended up having to put a `sleep 30` in my `start.sh` before starting the app and that seems to solve the problem. – lion_bash Nov 11 '18 at 03:21
The best practice would be to have a simple script (python would be ideal for you since the container already has everything you need) which tries to connect and validate the DB on a loop until successful. Sort of like the health check but in your app container instead. – Pramodh Valavala Nov 11 '18 at 05:28

score 0 · Answer 3 · answered Nov 10 '18 at 03:30

This is a common problem with multi containers. It is difficult to control the speed at which different containers start. Container orchestration solution like Kubernetes might be able to help you in such cases. Kubernetes has the concept of init containers which run to completion before your dependent container can start. You can find sample of init container here

https://www.handsonarchitect.com/2018/08/understand-kubernetes-object-init.html

This youtube vide might be helpful for you as well https://www.youtube.com/watch?v=n2FPsunhuFc

Docker compose getting mysql db fully started before flask app starts

3 Answers3

Linked