0

Machine - linux, 3.10.19 kernel

This is in a large distributed system, there are several servers and clients (on same as well as different nodes/machines) having TCP connections with each other.

Test case:
The client program node/machine is switched off (on purpose, test case) and the only way for server to know about his disconnection is via keepalive timer (idle time=40 sec, 4 probes, probe time=10 sec).

Good case:
This works fine in most of the cases, the server gets to know that the client has gone down in [40,70] sec.

Bad case:
But I am hitting another unique situation where while keepalive timer is running, the server tries sending some data to the client, and this in turn starts the TCP retransmission timer which overrides/kills the keepalive timer. It takes ~15 min for the retransmission timer to detect that the other end is not there anymore.

15 min is a lot of time for server to realize this. I am looking for ways how others handle such a situation. Do I need to tweak my retransmission timer values?

Thanks!

brokenfoot
  • 11,083
  • 10
  • 59
  • 80

2 Answers2

1

There is a completely separate configuration for retransmission timeouts.

From Linux's tcp.7 man page:

tcp_retries2 (integer; default: 15; since Linux 2.2)

The maximum number of times a TCP packet is retransmitted in established state before giving up. The default value is 15, which corresponds to a duration of approximately between 13 to 30 minutes, depending on the retransmission timeout. The RFC 1122 specified minimum limit of 100 seconds is typically deemed too short.

This is likely the value you'll want to adjust to change how long it takes to detect if your connection has vanished.

user207421
  • 305,947
  • 44
  • 307
  • 483
Brian White
  • 8,332
  • 2
  • 43
  • 67
  • Hi Brian, thanks for the reply. Any idea how such situations are dealt with? Putting such a short value on the retransmit timer doesn't seem like the way to go. IMO this is a very generic scenario and should be hit easily in applications. – brokenfoot Nov 09 '15 at 20:23
  • You have to understand that the primary design goal of TCP was _reliable_ communication, not _throughput_ or _low-latency_. It is assumed that the underlying network (IP, Ethernet, etc.) is unreliable and will have outages. Long retries are the way to accomplish this. Note that a _responsive_ host will "reset" the connection immediately if it is broken, not wait to timeout from multiple retransmissions. For a host to simply disappear is an exceptional condition so not a reason for optimization. – Brian White Nov 10 '15 at 15:34
0

I have the same issue with a linux kernel release 4.3.0-1-amd64:

  • I used a server and a client, connected to the same switch.

  • TCP keep-alive mecanism works correctly for client and server in the following cases:

    • When no message is sent between the cable disconnection and the socket disconnection (by the tcp keep-alive mecanism).
    • When the cable is disconnected between the client/server and the switch (which sets the link state to down) even if the client/server application tries to send a message.

  • When the wire is unpluged on the other side of the switch, TCP Keep-Alive frames are transmitted until an applicative message is sent. Then, TCP Retransmission frames are sent and TCP keep-alive frames stop being sent, which prevents the socket to be closed.

nulix
  • 1
  • 1