My flask_restx application is running in a docker container, with nginx as reverse proxy and Gunicorn workers.
When I send a HTTP-Request to my application, the expected result is a response after the application does its job, which takes about 3-5 minutes.
However, I fail to get a response to my request after about a minute after sending the request.
I added a timeout to my gunicorn config. Before I did that, I received a 502 Bad Gateway error from the nginx-server after 30 seconds, as the default timeout of gunicorn is 30 seconds.
In any case, in the background, my application works as intended and successfully does its job. It is only the response to my request that doesn't work properly.
Furthermore, I was able to see the print messages of my application when I didn't have --timeout 300 specified for gunicorn, but the workers timed out and booted up randomly during the time my application received a request.
Here is the gunicorn config as defined in the Dockerfile:
EXPOSE 5000
CMD gunicorn --workers 3 --bind 0.0.0.0:5000 wsgi:app --log-level=debug --timeout 300
The nginx config:
server {
listen 8080 ssl;
server_name localhost;
access_log off;
error_log off;
location / {
proxy_pass http://application-url:5000/;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $host;
proxy_redirect off;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 90001;
proxy_send_timeout 90001;
proxy_read_timeout 90001;
send_timeout 90001;
client_max_body_size 10m;
client_body_buffer_size 128k;
proxy_buffer_size 4k;
proxy_buffers 4 32k;
proxy_busy_buffers_size 64k;
}
}
Here is the log of the container:
Run pod=$( kubectl get pods -n application | grep application | grep -v nginx | awk '{print $1}')
[2022-08-03 16:23:16 +0000] [7] [DEBUG] Current configuration:
config: ./gunicorn.conf.py
wsgi_app: None
bind: ['0.0.0.0:5000']
backlog: 2048
workers: 3
worker_class: sync
threads: 1
worker_connections: 1000
max_requests: 0
max_requests_jitter: 0
timeout: 300
graceful_timeout: 300
keepalive: 2
limit_request_line: 4094
limit_request_fields: 100
limit_request_field_size: 8190
reload: False
reload_engine: auto
reload_extra_files: []
spew: False
check_config: False
print_config: False
preload_app: False
sendfile: None
reuse_port: False
chdir: /application/connector/
daemon: False
raw_env: []
pidfile: None
worker_tmp_dir: None
*** 101
group: 0
umask: 0
initgroups: False
tmp_upload_dir: None
secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
forwarded_allow_ips: ['127.0.0.1']
accesslog: None
disable_redirect_access_to_syslog: False
access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
errorlog: -
loglevel: debug
capture_output: False
logger_class: gunicorn.glogging.Logger
logconfig: None
logconfig_dict: {}
syslog_addr: udp://localhost:514
syslog: False
syslog_prefix: None
syslog_facility: user
enable_stdio_inheritance: False
statsd_host: None
dogstatsd_tags:
statsd_prefix:
proc_name: None
default_proc_name: wsgi:app
pythonpath: None
paste: None
on_starting: <function OnStarting.on_starting at 0x7f2c8dfc8a60>
on_reload: <function OnReload.on_reload at 0x7f2c8dfc8b80>
when_ready: <function WhenReady.when_ready at 0x7f2c8dfc8ca0>
pre_fork: <function Prefork.pre_fork at 0x7f2c8dfc8dc0>
post_fork: <function Postfork.post_fork at 0x7f2c8dfc8ee0>
post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f2c8ddb3040>
worker_int: <function WorkerInt.worker_int at 0x7f2c8ddb3160>
worker_abort: <function WorkerAbort.worker_abort at 0x7f2c8ddb3280>
pre_exec: <function PreExec.pre_exec at 0x7f2c8ddb33a0>
pre_request: <function PreRequest.pre_request at 0x7f2c8ddb34c0>
post_request: <function PostRequest.post_request at 0x7f2c8ddb3550>
child_exit: <function ChildExit.child_exit at 0x7f2c8ddb3670>
worker_exit: <function WorkerExit.worker_exit at 0x7f2c8ddb3790>
nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f2c8ddb38b0>
on_exit: <function OnExit.on_exit at 0x7f2c8ddb39d0>
proxy_protocol: False
proxy_allow_ips: ['127.0.0.1']
keyfile: None
certfile: None
ssl_version: 2
cert_reqs: 0
ca_certs: None
suppress_ragged_eofs: True
do_handshake_on_connect: False
ciphers: None
raw_paste_global_conf: []
strip_header_spaces: False
[2022-08-03 16:23:16 +0000] [7] [INFO] Starting gunicorn 20.1.0
[2022-08-03 16:23:16 +0000] [7] [DEBUG] Arbiter booted
[2022-08-03 16:23:16 +0000] [7] [INFO] Listening at: http://0.0.0.0:5000 (7)
[2022-08-03 16:23:16 +0000] [7] [INFO] Using worker: sync
[2022-08-03 16:23:16 +0000] [9] [INFO] Booting worker with pid: 9
[2022-08-03 16:23:16 +0000] [10] [INFO] Booting worker with pid: 10
[2022-08-03 16:23:16 +0000] [11] [INFO] Booting worker with pid: 11
[2022-08-03 16:23:16 +0000] [7] [DEBUG] 3 workers
[2022-08-03 16:23:34 +0000] [10] [DEBUG] GET /cgi-bin/ExportLogs.sh
[2022-08-03 16:23:39 +0000] [10] [DEBUG] POST /
[2022-08-03 16:23:39 +0000] [10] [DEBUG] GET /downloader.php
[2022-08-03 16:24:04 +0000] [10] [DEBUG] GET /
[2022-08-03 16:24:04 +0000] [10] [DEBUG] GET /swaggerui/droid-sans.css
[2022-08-03 16:25:16 +0000] [10] [DEBUG] Ignoring EPIPE
[2022-08-03 16:25:39 +0000] [10] [DEBUG] PUT /wp-content/plugins/w3-total-cache/pub/sns.php
[2022-08-03 16:26:01 +0000] [10] [DEBUG] GET /index.php
[2022-08-03 16:26:06 +0000] [10] [DEBUG] GET /listing/
[2022-08-03 16:26:42 +0000] [10] [DEBUG] GET /filter/jmol/js/jsmol/php/jsmol.php
[2022-08-03 16:27:50 +0000] [9] [DEBUG] POST /ViewPoint/admin/Site/ViewPointLogin
[2022-08-03 16:28:07 +0000] [10] [DEBUG] GET /wp-json/wp/v2/asked-question
[2022-08-03 16:28:57 +0000] [11] [DEBUG] GET /client/index.html
[2022-08-03 16:31:11 +0000] [11] [DEBUG] GET /_users/_all_docs
[2022-08-03 16:31:30 +0000] [11] [DEBUG] POST /json-rpc/
[2022-08-03 16:33:15 +0000] [9] [DEBUG] GET /wp-json/anycomment/v1/auth/wordpress
[2022-08-03 16:33:20 +0000] [11] [DEBUG] GET /sell-media-search/
[2022-08-03 16:33:21 +0000] [11] [DEBUG] GET /index.php
[2022-08-03 16:33:55 +0000] [10] [DEBUG] GET /module/smartblog/archive
[2022-08-03 16:34:12 +0000] [9] [DEBUG] GET /.git/config
[2022-08-03 16:34:22 +0000] [10] [DEBUG] GET /docker-compose.yml
[2022-08-03 16:34:26 +0000] [9] [DEBUG] GET /docker-compose.prod.yml
[2022-08-03 16:34:30 +0000] [10] [DEBUG] GET /docker-compose.production.yml
[2022-08-03 16:34:34 +0000] [10] [DEBUG] GET /docker-compose.staging.yml
[2022-08-03 16:34:38 +0000] [10] [DEBUG] GET /docker-compose.dev.yml
[2022-08-03 16:34:42 +0000] [9] [DEBUG] GET /docker-compose-dev.yml
[2022-08-03 16:34:46 +0000] [11] [DEBUG] GET /docker-compose.override.yml
[2022-08-03 16:34:49 +0000] [9] [DEBUG] GET /docker-cloud.yml
[2022-08-03 16:35:47 +0000] [9] [DEBUG] GET /maint/modules/endpointcfg/endpointcfg.php
