3

Two applications built with flask API, trying to receive a response with huge JSON response fails with Error 10054, 'An existing connection was forcibly closed by the remote host'

I could narrow the issue that when response is huge it fails

@api.route('/endpoint', methods=['POST'])
def endpoint():

   result = {small / huge dict}

   return jsonify({'result': result}), 200

caller side:

result = requests.post(url, params=data['args'], json=data['payload'])
        return result.json()['result']

Error log:

File "C:\Program Files (x86)\Python36-32\lib\http\client.py", line 1331, in getresponse
response.begin()
File "C:\Program Files (x86)\Python36-32\lib\http\client.py", line 321, in begin
self.headers = self.msg = parse_headers(self.fp)
File "C:\Program Files (x86)\Python36-32\lib\http\client.py", line 206, in parse_headers
line = fp.readline(_MAXLINE + 1)
File "C:\Program Files (x86)\Python36-32\lib\socket.py", line 586, in readinto
return self._sock.recv_into(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

Update:

tried to yield response as plan text and failures where much less, but still once in a while the issue appears.

def response(output):
    return Response(response_generator(output), mimetype='text/plain')


def response_generator(result):
    result_str = json.dumps(result)
    for row in [result_str[i:i + 1024*1024] for i in range(0, len(result_str), 1024*1024)]:
         yield row
seismael
  • 77
  • 1
  • 10
  • 1
    your server is configured to timeout before the full response can be processed. How to increase that timeout depends on what server you are using. This assumes that you have already tried improving the performance of the endpoint itself. – avigil Mar 09 '18 at 06:47
  • @avigil i noticed this issue repeated with many reasons ans solutions, tried out many of them including code improvements and server changes, i use the built in flask server, now i am trying to ask for a different approach, like yielding response, text instead of json, separate the endpoint like notrmal and ftp parts.... – seismael Mar 09 '18 at 07:14
  • maybe see https://stackoverflow.com/questions/24127601/uwsgi-request-timeout-in-python (uwsgi is the default server used by flask) – avigil Mar 09 '18 at 07:17

1 Answers1

-1

As the Error log, the problems maybe one of below:

  1. You are trying to open a url twice in your code. I don't see the whole caller side, so i cannot say if it's true or provide any further solution for sure

  2. The request return bytes, which need to be decoded or Json decoding fail. I suggest you should change the code as below:

    result = requests.post(url, params=data['args'], json=data['payload']).text return result

If you need work with json, you should use: 

result = json.loads(requests.post(url, params=data['args'], json=data['payload']).text)
Minh Tuấn Nguyễn
  • 48
  • 3
  • this wouldn't really explain why they only encounter the error for large response objects, and there is no reason not to use the built in json deserializer provided by the `requests` package – avigil Mar 09 '18 at 07:18
  • Hi @avigil , i don't leave any comment about json deserializer provided by `requests` package. I use `json.loads` for loading huge json stock data everyday. So, from my experiences, i suggest json.loads for the answer. It's an opinion – Minh Tuấn Nguyễn Mar 09 '18 at 07:34
  • fair enough. [Response.json()](https://github.com/requests/requests/blob/master/requests/models.py#L868-L892) actually uses `json` module underneath, but adds compatibility for python <2.4 (when `json` was added to standard library) and also handles different encodings for you, so it will almost always be the better choice. Anyway the problem here is probably on the server side, not on the client. – avigil Mar 09 '18 at 07:43