Connection aborted: RemoteDisconnected ('Remote end closed connection without response') after approx 5 mins of inactivity when sending POST

Question

I have the idle timeout set to 60 minutes. However, tests with a Python client talking to a Python server, the connection resets by peer approx 5 mins of inactivity when sending POST. The activity is still happening on the server side even after connection is reset or disconnected.

import requests
import logging
import flask
from flask import Flask

app = Flask(__name__)
logging.basicConfig(level=logging.INFO, format=f'%(asctime)s %(levelname)s %(name)s %(threadName)s : %(message)s')

@app.before_first_request
def before_first_request():
    app.logger.info("before_first_request")

@app.before_request
def before_request():
    app.logger.info("before_request")

@app.route('/run', methods=['POST'])
def run(redirect_url="",additional_testng_parameters=""):
    url = "http://192.168.1.8:7070/" + "run"
    app.logger.info("url is:" + url)
   
    session = requests.Session()   
    app.logger.info("Request placed for " + url)
    response = "Text"
    try:
        response = session.post(url=url, timeout=3600)
        resultdata = response.headers['Content-Type']
        app.logger.info("Response received for : " + url + " is:" + str(response.status_code))
        app.logger.info("Response received for : " + url + " is:" + str(resultdata))
        response = make_response("Execution is successful")
    except Exception as err:
        app.logger.error(err)
    return response

@app.errorhandler(Exception)
def server_error(err):
    app.logger.exception(err)
    return "There is some error on Test Invoker side", 500

if __name__ == '__main__':
    from waitress import serve
    serve(app, host="0.0.0.0", port=8080)

Roughly after 5 minutes getting below error message

ERROR app waitress-0 : ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Below are few references but it does not resolve my problem

requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

https://www.pythonfixing.com/2022/01/fixed-aborted-remotedisconnected-end.html

RESOLVED Below piece of code helped me to resolve the problem

import socket
from urllib3.connection import HTTPConnection

HTTPConnection.default_socket_options = (
    HTTPConnection.default_socket_options + [
        (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
        (socket.SOL_TCP, socket.TCP_KEEPIDLE, 45),
        (socket.SOL_TCP, socket.TCP_KEEPINTVL, 10),
        (socket.SOL_TCP, socket.TCP_KEEPCNT, 6)
    ]
)

Is the connection passing through a proxy/router? If so, that likely has its own idle timeout handling that is killing the connection before your timeout elapses. Have you tried enabling TCP keep-alives on the connection? — Remy Lebeau, May 26 '22 at 19:16
No, the connection is not passing through proxy/router. Yes I tried tried enabling TCP keep-alive but it didn't worked. — hardy_sandy, May 27 '22 at 06:47
Below piece of code helped me to resolve the problem: import socket from urllib3.connection import HTTPConnection HTTPConnection.default_socket_options = ( HTTPConnection.default_socket_options + [ (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1), (socket.SOL_TCP, socket.TCP_KEEPIDLE, 45), (socket.SOL_TCP, socket.TCP_KEEPINTVL, 10), (socket.SOL_TCP, socket.TCP_KEEPCNT, 6) ] ) — hardy_sandy, Jun 04 '22 at 04:14

score 4 · Answer 1 · answered Jun 07 '22 at 08:02

Try this:

import socket
from urllib3.connection import HTTPConnection

HTTPConnection.default_socket_options = (
    HTTPConnection.default_socket_options + [
        (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
        (socket.SOL_TCP, socket.TCP_KEEPIDLE, 45),
        (socket.SOL_TCP, socket.TCP_KEEPINTVL, 10),
        (socket.SOL_TCP, socket.TCP_KEEPCNT, 6)
    ]
)

score 0 · Answer 2 · answered Dec 15 '22 at 19:54

0

I experienced this error while using Gunicorn server for fastapi. I solved it by setting '--timeout' to a value greater than 30.

answered Dec 15 '22 at 19:54

Jide

9
2

score 0 · Answer 3 · answered Apr 10 '23 at 22:15

I found Voidy's solution to work but modified it like so:


class KeepAliveTransport(sentry_sdk.HttpTransport):

    def _get_pool_options(self, ca_certs):
        opts = super()._get_pool_options(ca_certs)
        opts["socket_options"] = urllib3.connection.HTTPConnection.default_socket_options + [
            (socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1),
            (socket.SOL_TCP, socket.TCP_KEEPIDLE, 45),
            (socket.SOL_TCP, socket.TCP_KEEPINTVL, 10),
            (socket.SOL_TCP, socket.TCP_KEEPCNT, 6),
        ]
        return opts

sentry_sdk.init(transport=KeepAliveTransport)

Connection aborted: RemoteDisconnected ('Remote end closed connection without response') after approx 5 mins of inactivity when sending POST

3 Answers3