16

I have an nginx-gunicorn-flask setup running on google compute and I keep getting the following from gunicorn:

[2019-04-19 20:50:49 +0000] [3345] [DEBUG] POST /
[2019-04-19 20:50:49 +0000] [3345] [DEBUG] Ignoring EPIPE

out of 100 requests about 23 come out like that. In the access log only 23 requests show and they're all 200.

from nginx access logs show a 504 and in the error logs I see this:

2019/04/19 20:50:49 [error] 3097#3097: *295 upstream timed out (110: Connection timed out) while sending request to upstream, client: ip, server: , request: "POST / HTTP/1.1", upstream: "http://unix:/home/user/Server/server.sock/", host: "ip"

I tried setting 

proxy_connect_timeout 75s;
proxy_read_timeout 300s;

on the location on nginx and --timeout 300 in gunicorn based on other questions but it didn't help.

The message from gunicorn isn't really helping determine the cause and I didn't find any info about it

Been trying to solve this for a while and I'd appreciate any ideas. Also each of these requests take about 1-2 seconds and jmeter shows a huge latency until the results show up.

Looking at gunicorn's code this is a errno.EPIPE exception having to do with the socket...

K41F4r
  • 1,443
  • 1
  • 16
  • 36
  • i have same issue. i think it's due to some execution. – Devang Hingu Dec 03 '20 at 07:29
  • Do you host Nginx and app server on the same host? Can you show your Nginx and Gunicorn config? – HTF Mar 24 '21 at 09:40
  • somewhat related (on the topic of services and sigpipe): [Hudson : “yes: standard output: Broken pipe”](https://stackoverflow.com/q/20573282/4279) – jfs Mar 24 '21 at 17:46
  • im getting those log messages when workers are restarted and take some time to come up, while health-checks are running against nginx. Im only guessing, but it could be that gunicorn worker-manager queues up requests during worker restart, then worker processes the requests but the client connection is already closed when the response is ready. As said, just a guess. The messages disappear when workers are up and waiting. – leberknecht Aug 08 '23 at 13:54

3 Answers3

4

I faced similar issues and timeout error, in addition with increasing timeout value, add Nginx config proxy_ignore_client_abort on; check this thread and Nginx doc

proxy_ignore_client_abort on;
Dharman
  • 30,962
  • 25
  • 85
  • 135
Rathish Kumar B
  • 1,271
  • 10
  • 21
0

If Gunicorn and NGINX are running on the same server, this could be a permission issue.

Check which user the Gunicorn server is running your application as with ps aux | grep gunicorn, then you can change the permissions accordingly.

Stop NGINX, and run the commands below

chown gunicorn_user_here.nginx /var/lib/nginx/tmp/proxy -v
rm /var/lib/nginx/tmp/proxy/* -R
chmod 777 /var/lib/nginx/tmp -v

Start NGINX and check the logs.

Paulo Marques
  • 775
  • 4
  • 15
-2

In my case, the hard drive was full and that was causing the issue.

Fonnae
  • 453
  • 1
  • 4
  • 12