10

I have a Flask application that I run through gunicorn (on a Dell latitude 5410 laptop with Ubuntu 20.04) and I configured the Docker to run it as suggested in this wonderful guide.

Here is my boot.sh:

#!/bin/sh
source venv/bin/activate
while true; do
    flask db init
    flask db migrate
    flask db upgrade
    if [[ "$?" == "0" ]]; then
        break
    fi
    echo Deploy command failed, retrying in 5 secs...
    sleep 5
done
exec gunicorn -b 0.0.0.0:5000 --access-logfile - --error-logfile - main:app

and the Dockerfile just call it as entrypoint:

FROM python:3.8-alpine

RUN adduser -D diagnosticator

RUN apk add --no-cache bash mariadb-dev mariadb-client python3-dev build-base libffi-dev openssl-dev

WORKDIR /home/diagnosticator

COPY requirements.txt requirements.txt
RUN python -m venv venv
RUN venv/bin/pip install -U pip
RUN venv/bin/pip install wheel
RUN venv/bin/pip install -r requirements.txt
RUN venv/bin/pip install gunicorn pymysql

COPY app app
COPY upload upload
COPY variant_dependencies variant_dependencies
COPY main.py config.py boot.sh ./
COPY convert_VCF_REDIS.py asilo_variant_functions.py cloud_bigtable_functions.py mongodb_functions.py redis_functions.py ./
COPY docker_functions.py ./
RUN chmod a+x boot.sh

ENV FLASK_APP main.py

RUN chown -R diagnosticator:diagnosticator ./
USER diagnosticator

EXPOSE 5000
ENTRYPOINT ["./boot.sh"]

I run the Docker in a docker-network with:

docker network create --driver bridge diagnosticator-net

The problem I noticed is that the Docker container for some reason hangs after a bit of inactivity with gunicorn error:

[CRITICAL] WORKER TIMEOUT

here the docker logs:

192.168.32.1 - - [12/Jun/2021:12:28:30 +0000] "GET /patient_page/GHARIgANY21uuITW2196372 HTTP/1.1" 200 19198 "http://127.0.0.1:3000/patient_result" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.77 Safari/537.36"
[2021-06-12 12:29:10 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:16)
[2021-06-12 12:29:10 +0000] [16] [INFO] Worker exiting (pid: 16)
[2021-06-12 12:29:10 +0000] [17] [INFO] Booting worker with pid: 17
[2021-06-12 12:29:10,905] INFO in __init__: Diagnosticator-local startup
[2021-06-12 12:29:41 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:17)
20
[2021-06-12 12:29:41 +0000] [17] [INFO] Worker exiting (pid: 17)
[2021-06-12 12:29:41 +0000] [18] [INFO] Booting worker with pid: 18
[2021-06-12 12:29:42,094] INFO in __init__: Diagnosticator-local startup
[2021-06-12 12:30:12 +0000] [1] [CRITICAL] WORKER TIMEOUT (pid:18)
[2021-06-12 12:30:12 +0000] [18] [INFO] Worker exiting (pid: 18)

and the application just hangs loading forever any page you try to access.

I've seen some suggestions here and googling it, like this one or this one, suggesting to work on gunicorn --timeout but the problem keeps happening no matter what I put there.

Any help will be deeply appreciated cause I cannot figure out any real solution!

cccnrc
  • 1,195
  • 11
  • 27
  • Are you querying a database? is the connectivity properly established from docker container to the database ? – Kaustubh Desai Jun 14 '21 at 13:59
  • @KaustubhDesai do you mean if the docker application connects to a DB? How can this affect `gunicorn`? – cccnrc Jun 14 '21 at 14:31
  • If the docker application connects to a database and if the connection is not established because of port blocking, container not on the same docker network, etc the worker while connecting to DB will always time out. – Kaustubh Desai Jun 14 '21 at 14:59
  • Are you able to run this using Flask's [development server](https://flask.palletsprojects.com/en/2.0.x/server/)? If yes, we can investigate `gunicorn` further, else we may have to investigate the application itself. – hexbioc Jun 16 '21 at 19:34
  • 1
    This is likely unrelated to gunicorn, the timeout is more likely happening in the app. If you can run the container locally and enable debugging you should be able to see some traceback and identify what's hanging your app. My guess is you have a local db running and the docker container can't see it because you didn't forward the ports. – AArias Jun 17 '21 at 09:17
  • You will have to provide more information about the code in your app route. If you use the builtin development server, do you have the same problem? If so, it's probably not a gunicorn issue. – sytech Jun 17 '21 at 17:38
  • how about checking your ram while you start this container?? just monitor that for a while. – Deep Bhatt Jun 18 '21 at 21:16
  • Can you share debug logs by using the debug flag "--log-level debug" when starting gunicorn? – pyronic Jun 19 '21 at 12:12
  • I think this discussion might helps you. https://github.com/benoitc/gunicorn/issues/1801 – Yz. Jun 21 '21 at 01:58
  • @cccnrc if your problem is still not resolved you can check [this](https://stackoverflow.com/questions/10855197/gunicorn-worker-timeout-error) post – Chandan Jul 02 '21 at 05:59

1 Answers1

0

Try to run Gunicorn with --log-level debug... It should give some trace about the error.

Plus, you can try to add --worker-class in your gunicorn command

avimimoun
  • 799
  • 9
  • 23