164

I'm trying to figure out why my app's TCP/IP connection keeps hiccuping every 10 minutes (exactly, within 1-2 seconds). I ran Wireshark and discovered that after 10 minutes of inactivity the other end is sending a packet with the reset (RST) flag set. A google search tells me "the RESET flag signifies that the receiver has become confused and so wants to abort the connection" but that is a little short of the detail I need. What could be causing this? And is it possible that some router along the way is responsible for it or would this always come from the other endpoint?

Edit: There is a router (specifically a Linksys WRT-54G) sitting between my computer and the other endpoint -- is there anything I should look for in the router settings?

Luke
  • 18,585
  • 24
  • 87
  • 110
  • 18
    Here's another: Comcast – Tom Ritter Oct 30 '08 at 18:44
  • 2
    Heh luckily I don't have a dependency on Comcast as this is occurring within a LAN. I wish I could shift the blame that easily tho ;) – Luke Oct 30 '08 at 18:46
  • Did you ever get this figured out? I can't comment because I don't have enough points, but I have the same exact problem you were having and I am looking for a fix. –  Jan 13 '11 at 21:15
  • 1
    What service this particular case refers to? It may be possible to set keepalive on the socket (from the app-level) so long idle periods don't result in someone (in the middle or not) trying to force a connection reset for lack of resources. – arielf Jun 03 '16 at 23:23
  • 2
    "Comcast" you say? :D Check out this related repo: https://github.com/tylertreat/comcast – joonas.fi Jan 17 '17 at 11:33

11 Answers11

103

A 'router' could be doing anything - particularly NAT, which might involve any amount of bug-ridden messing with traffic...

One reason a device will send a RST is in response to receiving a packet for a closed socket.

It's hard to give a firm but general answer, because every possible perversion has been visited on TCP since its inception, and all sorts of people might be inserting RSTs in an attempt to block traffic. (Some 'national firewalls' work like this, for example.)

Will Dean
  • 39,055
  • 11
  • 90
  • 118
29

Run a packet sniffer (e.g., Wireshark) also on the peer to see whether it's the peer who's sending the RST or someone in the middle.

Alexander
  • 9,302
  • 2
  • 26
  • 22
17

One thing to be aware of is that many Linux netfilter firewalls are misconfigured.

If you have something like:

-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT 

-A FORWARD -p tcp -j REJECT --reject-with tcp-reset

then packet reordering can result in the firewall considering the packets invalid and thus generating resets which will then break otherwise healthy connections.

Reordering is particularly likely with a wireless network.

This should instead be:

-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT 

-A FORWARD -m state --state INVALID -j DROP 

-A FORWARD -p tcp -j REJECT --reject-with tcp-reset

Basically anytime you have:

... -m state --state RELATED,ESTABLISHED -j ACCEPT

it should immediately be followed by:

... -m state --state INVALID -j DROP

It's better to drop a packet then to generate a potentially protocol disrupting tcp reset. Resets are better when they're provably the correct thing to send... since this eliminates timeouts. But if there's any chance they're invalid then they can cause this sort of pain.

wes
  • 7,795
  • 6
  • 31
  • 41
MaZe
  • 356
  • 2
  • 4
  • 1
    Oh my god man, thank you so much for this! You fixed my firewall! I've been looking for a solution for days. In my case I was using NetworkManager with "ipv4.method = shared" and had to apply this fix to my upstream interface which had the restrictive iptables rules on it. How or where exactly did you learn of this? – David Apr 12 '22 at 20:55
16

I've just spent quite some time troubleshooting this very problem. None of the proposed solutions worked. Turned out that our sysadmin by mistake assigned the same static IP to two unrelated servers belonging to different groups, but sitting on the same network. The end results were intermittently dropped vnc connections, browser that had to be refreshed several times to fetch the web page, and other strange things.

OutputLogic
  • 386
  • 4
  • 11
9

RST is sent by the side doing the active close because it is the side which sends the last ACK. So if it receives FIN from the side doing the passive close in a wrong state, it sends a RST packet which indicates other side that an error has occured.

Vishal gupta
  • 107
  • 1
  • 1
  • 7
    Both sides send and receive a FIN in a normal closure. There is nothing wrong with this situation, and therefore no reason for one side to issue a reset. The first sentence doesn't even make sense. – user207421 Jan 14 '16 at 05:13
  • 3
    [RST, ACK] can also be sent by the side receiving a SYN on a port not being listened to. In a case I ran across, the RST/ACK came about 60 seconds after the first SYN. FWIW – Les Mar 22 '17 at 16:05
  • @MarquisofLorne, the first sentence itself may be treated as incorrect. But the phrase "in a wrong state" in second sentence makes it somehow valid. – Victor Yarema Aug 06 '20 at 15:33
9

Here are some cases where a TCP reset could be sent.

  • Non-Existence TCP endpoint: The client sends SYN to a non-existing TCP port or IP on the server-side. The server will send a reset to the client.
  • SYN matches the existing TCP endpoint: The client sends SYN to an existing TCP endpoint, which means the same 5-tuple. The server will send a reset to the client.
  • Accept Queue Full: When the accept queue is full on the server-side, and tcp_abort_on_overflow is set. The server will send a reset to the client.
  • Half-Open Connections: When the server restarts itself. Then all connections before would receive reset from server side.
  • Firewall: The firewall could send a reset to the client or server
  • Time-Wait Assassination: When the client in the time-wait state, receives a message from the server-side, the client will send a reset to the server.
  • Aborting Connection: When the client aborts the connection, it could send a reset to the server
zangw
  • 43,869
  • 19
  • 177
  • 214
8

If there is a router doing NAT, especially a low end router with few resources, it will age the oldest TCP sessions first. To do this it sets the RST flag in the packet that effectively tells the receiving station to (very ungracefully) close the connection. this is done to save resources.

MKII
  • 892
  • 11
  • 36
7

Some firewalls do that if a connection is idle for x number of minutes. Some ISPs set their routers to do that for various reasons as well.

In this day and age, you'll need to gracefully handle (re-establish as needed) that condition.

Brian Knoblauch
  • 20,639
  • 15
  • 57
  • 92
  • 2
    The connection is re-established just fine, the problem is that the brief period of disconnect causes an alert unnecessarily. – Luke Oct 30 '08 at 18:41
  • 1
    I've had problems specifically with Cisco PIX/ASA equipment. They have especially short timeouts as defaults. The cheaper equipment is usually "better" in this regard (as in they don't timeout real fast)... – Brian Knoblauch Oct 30 '08 at 18:54
  • 2
    I would even add that TCP was never actually completely reliable from persistent connections point of view. It just becomes more noticeable from time to time. Another interesting example: some people may implement logic that marks a TCP client as offline as soon as connection closure or reset is being detected. And then sometimes they don't bother to give a client a chance to reconnect. This is obviously not completely correct. – Victor Yarema Aug 06 '20 at 15:38
2

This is because there is another process in the network sending RST to your TCP connection.

Normally RST would be sent in the following case

  • A process close the socket when socket using SO_LINGER option is enabled
  • OS is doing the resource cleanup when your process exit without closing socket.

In your case, it sounds like a process is connecting your connection(IP + port) and keeps sending RST after establish the connection.

Ben
  • 179
  • 1
  • 3
1

In most applications, the socket connection has a timeout. If there is no communication between the client and the server within the timeout, the connection is reset as you observe. A great example is a FTP server, if you connect to the server and just leave the connection without browsing or downloading files, the server will kick you off the connection, usually to allow other to be able to connect. I guess this is what you are experiencing with your connection. So take a look in the server application, if that is where you get the reset from, and see if it indeed has a timeout set for the connection in the source code.

nivs1978
  • 1,126
  • 14
  • 20
0

Just had a case. Client1 connected to Server. They are sending data via websocket protocol and the TCP connection is kept alived. Server is python flask and listening on Port 5000. Then Client2(same IP address as Client1) send a HTTP request to Server. Now if you interrupt Client1 to make it quit. Then a "connection reset by peer 104" happens in Server side and Client2.

Simply put, the previous connection is not safely closed and a request is sent immediately for a 3 way handshake. The Server side got confused and sent a RST message.

马卓群
  • 1
  • 1
  • 1
    What you described doesn't seem like the normal behavior for a server. Is there a reason why the second connection was RST to client 2, simply because client one quit? (And by quit, I assume you mean the client program shut down the connection in an orderly fashion). – benc Mar 07 '23 at 03:53