0

I have an AWS server (medium) running in EU West, and there are roughly 250 devices connected but are also always reconnecting due to internet connectivity issues, but for some reason, the amount of TCP connections to the server grows until it reaches around 4300. Then no new connections are allowed to the server. I have confirmed that it is isolated to WebSocket requests and not regular HTTP requests.

WebSocket connections are kept per device in a Map with device UUID as key; it sometimes happens that a device will send a request for a new WS connection even though the server has a connection to the device. In this case, the current connection is closed, and an error is returned so that the device can retry the connection request.

Below is the code snippet from the Controller handling the connections using LegacyWebSocket. Connections are closed using out.close() as per https://www.playframework.com/documentation/2.5.x/JavaWebSockets#handling-websockets-using-callbacks 

public LegacyWebSocket<String> create(String uuid) {
    logger.debug("NEW WebSocket request from {}, creating new socket...", uuid);

    if(webSocketMap.containsKey(uuid)){
        logger.debug("WebSocket already exists for {}, closing existing connection", uuid);

        webSocketMap.get(uuid).close();

        logger.debug("Responding forbidden to force WS restart from device {}", uuid);
        return WebSocket.reject(forbidden());
    }

    LegacyWebSocket<String> ws = WebSocket.whenReady((in, out) -> {
        logger.debug("Adding downstream connection to webSocketMap-> {} webSocketMap.size() = {}",uuid, webSocketMap.size());
        webSocketMap.put(uuid,out);
        // For each event received on the socket,
        in.onMessage(message->{

            if(message.equals("ping")){
                logger.debug("PING received from {} {}",uuid, message);
                out.write("pong");
            }
        });


        // When the socket is closed.
        in.onClose(() -> {
            logger.debug("onClose, removing for {}",uuid);

            webSocketMap.remove(uuid);
        });
    });

    return ws;
}

How can I ensure that Play Framework closes the TCP connection for closed WS connections?

The call that I use to check the amount of TCP connections is netstat -n -t | wc -l

Loïc Faure-Lacroix
  • 13,220
  • 6
  • 67
  • 99
nicolasventer
  • 1
  • 1

1 Answers1

0

Looks like a TCP keep-alive issue - i.e. that the TCP connections become stale because of connectivity issues on the client side and the server does not handle or clean up the stale connections in time before the limit is reached.

This link will help you configure the TCP keep-alive on your server to ensure that the stale connections are cleaned up in time.

Community
  • 1
  • 1
mdw
  • 21
  • 3