SIGPIPE writing to a closed pipe error in an EBS Docker app

Question

I have an inexplicable error when running a docker container in EBS vs locally. The container runs a naked uWSGI process that loads an app that runs long-running (over 5 seconds) requests. The docker container runs fine locally; and the entire code path runs fine on the EBS host when run through ssh on the box (by submitting requests programmatically to entry function, that emulates the POST parameters from the client). But when the code path is called over http via an API route, it errors out with:

Fri May  7 03:01:40 2021 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /preview-map (ip 172.17.0.1) !!!
2021-05-07 03:01:40] log_exception 1761 - Exception on /preview-map [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2292, in wsgi_app
    response = self.full_dispatch_request()
   ...
  File "./scene.py", line 135, in scenePreview
    quality=10)  # quality=1 => no shadow/reflection, quality=10 is 'normal'
  File "/usr/local/lib/python3.7/site-packages/vapory/vapory.py", line 102, in render
    quality, antialiasing, remove_temp)
  File "/usr/local/lib/python3.7/site-packages/vapory/io.py", line 114, in render_povstring
    raise IOError("POVRay rendering failed with the following error: "+err)
TypeError: can only concatenate str (not "bytes") to str

I debugged for hours, assuming that the error is about the stack trace: POVRay (a 3D scene rendering engine abstracted in Python by the vapory library), but I can run the POVRay function on the EBS docker container no problem.

Is it possible that my uWSGI process in EBS is closing my connection after 3 seconds and this break in the code is just a co-incidence? It's docker, so there really can't be any difference between my local and remote environments other than something EBS is doing to close connections prematurely.

I am stumped. Any idea how to debug this further?

score 1 · Answer 1 · answered May 11 '21 at 17:19

1

Same solution as here: Fixing broken pipe error in uWSGI with Python

Increase uWSGI timeouts.

ENV UWSGI_CONNECT_TIMEOUT=2400 UWSGI_READ_TIMEOUT=2400 UWSGI_SEND_TIMEOUT=2400

answered May 11 '21 at 17:19

metalaureate

7,572
9
54
93

score 0 · Answer 2 · answered May 16 '21 at 13:01

TL;DR - The err in the /usr/local/lib/python3.7/site-packages/vapory/io.py file should be err.decode('ascii'). This is likely a bug that got fixed in later versions.

Lets have a look at the last part of the traceback:

  File "/usr/local/lib/python3.7/site-packages/vapory/io.py", line 114, in render_povstring
    raise IOError("POVRay rendering failed with the following error: "+err)
TypeError: can only concatenate str (not "bytes") to str

So apparently, the error is coming from the render_povstring function defined somewhere in the /usr/local/lib/python3.7/site-packages/vapory/io.py file at this line:

raise IOError("POVRay rendering failed with the following error: "+err)

It appears that err is of type bytes rather than str, and to prove it, I installed the vapory module and navigated to the io.py file. What I see in the file in the render_povstring function is

raise IOError("POVRay rendering failed with the following error: "+err.decode('ascii'))

So this used to be a bug in the package that got fixed likely in later versions.

The error was a red herring. The server was closing the process because of uWSGI timeouts. When I migrated the process to job queue (Django Q FTW), the problem went away. — metalaureate, May 16 '21 at 17:16

SIGPIPE writing to a closed pipe error in an EBS Docker app

2 Answers2