3

Background:

I am running a TCP server on one machine with multiple clients on separate machines connecting over TCP, and I am monitoring network traffic with Wireshark, as well as with logging from within my server application, and through the System.Diagnostics tracelistener on System.Net.Sockets in verbose mode.

Problem:

Prompted to examine my logs due to some unexpected disconnects, I see some very strange behavior. According to the server application logs, and the System.Diagnostic output logs, my server is sending a 4 byte packet to a client using begin/end send. The BeginSend completes, and the EndSend also completes saying that it successfully sent the 4 byte packet.

However, when I look at the Wireshark logs, that packet never shows up. I am running Wireshark on the server machine, so there should be no reason for the packet to show up in my server and trace logs, but not in the Wireshark logs on the same machine.

Also, there is an unexpected disconnect that occurs soon after the supposedly successful packet send (~30 seconds later), which is caused by a SocketException on my server's EndReceive method. But during the time between the attempted send from the server, the server is acknowledging packets received from the client, so I know the connection is still active.

Has anyone out there had a similar experience, or know of a bug or something that might be causing this?

I would hate to think that this is occurring at the socket level, where TCP is saying that my packet is sent when it never even made it onto the wire, which would mean that I cannot rely on TCP as a reliable transport (which of course is the whole point of TCP).

Log Samples

From my server application:

    2011-09-07 10:41:38,812 Attempting to send Packet (BeginSend - 4 bytes)
    2011-09-07 10:41:38,812 Sent Packet (EndSend - 4 bytes)

From the System.Diagnostics trace log:

System.Net.Sockets Verbose: 0 : [4376] Socket#19699911::BeginSend()
    DateTime=2011-09-07T17:41:38.8125000Z
System.Net.Sockets Verbose: 0 : [0980] Data from Socket#19699911::PostCompletion
    DateTime=2011-09-07T17:41:38.8125000Z
System.Net.Sockets Verbose: 0 : [0980] 00000000 : 02 04 00 00                                     : ....
    DateTime=2011-09-07T17:41:38.8125000Z
System.Net.Sockets Verbose: 0 : [4376] Socket#19699911::EndSend(OverlappedAsyncResult#44209720)
    DateTime=2011-09-07T17:41:38.8125000Z
System.Net.Sockets Verbose: 0 : [4376] Exiting Socket#19699911::EndSend()   -> 4#4
    DateTime=2011-09-07T17:41:38.8125000Z
System.Net.Sockets Verbose: 0 : [4376] Exiting Socket#19699911::BeginSend()     -> OverlappedAsyncResult#44209720
    DateTime=2011-09-07T17:41:38.8125000Z

I would paste the Wireshark logs, too, but essentially there is nothing registering on the interface for that device at that time, except for the packets coming from the client, and the corresponding acknowledgments from the server.

Edit: As requested, here is the code for the send (shortened for space, and other reasons). Pretty simple, not much in the way of anything that could go wrong.

In my BeginSend method:

socket.BeginSend(data, 0, data.Length, SocketFlags.None, EndSend, state);

In my EndSend method:

bytesSent = socket.EndSend(ar);

Note: This is not my first rodeo, as they say... I have been writing servers and clients using TCP sockets for the last 15 years, and have never experienced this before.

Also, the version of .NET that I am using is 4.0... if that is of any relevance.

Help!

Aron
  • 372
  • 5
  • 18
  • Can you add the relevant client & server code to the post? Maybe there's a mistake in the code. – Jacob Sep 07 '11 at 19:02
  • Added some code... but really, there's nothing special going on. It's a simple BeginSend and EndSend... you could take any sample code from MSDN or elsewhere and reproduce what I'm seeing. – Aron Sep 07 '11 at 19:30
  • Maybe my understanding of how TCP works from the .NET platform is incorrect... would a Socket.EndSend call return successfully when the message has been transmitted to, and acknowledged by, the receiving party? Or would the EndSend be successful when .NET has sent the message successfully to the network interface? If it is the latter, a faulty network adapter might explain what I am seeing. – Aron Sep 07 '11 at 19:46
  • Does the client see that 4-byte transmission? Your post implies that the conversation between client and server continues regardless. – anton.burger Sep 07 '11 at 20:20
  • Also, what details does the `SocketException` give you? Message? [Error code](http://msdn.microsoft.com/en-us/library/system.net.sockets.socketexception.errorcode.aspx)? – anton.burger Sep 07 '11 at 20:23
  • The client does not see the 4 byte packet I am sending. The packet never even makes it to the adapter layer, or I would probably see something logged in Wireshark. The SocketException is as follows: System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond – Aron Sep 07 '11 at 20:34
  • I see this same behavior under framework 4.5.x and 4.6.x, small packets seem to never make it over the network while all code surrounding communications is reporting successful sends. I've tried modifying a number of socket options. When using an unmanaged (C++) test tool which executes identical code (functionally speaking) all data is transmitted in an expedient fashion. If I "blast" the socket with many small writes (which breaks our protocol) each send goes through in a timely fashion. Ridiculous. – Shaun Wilson Mar 22 '16 at 22:36

4 Answers4

0

Well this question was ages ago, so you've probably moved on... in case someone else has the same problem, you may want to check whether Windows' Memory Pressure Protection might be kicking in. This will cause packets to be dropped when there are multiple simultaneous connections, which is what you're seeing.

Another experiment you can try is to temporarily turn off the firewall on your server machine. This will also turn off all of these protection mechanisms (if I'm not mistaken) and if things work then, you know that one of those is the culprit.

redtuna
  • 4,586
  • 21
  • 35
0

One issue I've encountered with socket programming in .NET was related to garbage collection. My code was creating a socket and assigning to a local variable in my method instead of assigning to a member variable. The socket was then garbage-collected, causing a SocketException. Apparently, doing a BeginSend wasn't enough to hold a reference to that socket.

Moving the socket to a member variable solved the issue for me. Hopefully this is relevant to the issue you're encountering.

Jacob
  • 77,566
  • 24
  • 149
  • 228
  • Unfortunately this is not the issue... but the comment is still relevant for socket programming. Any socket interaction that spans multiple threads or methods should be referencing a variable that isn't going to get collected like that. – Aron Sep 07 '11 at 20:16
0

I would hate to think that this is occurring at the socket level, where TCP is saying that my packet is sent when it never even made it onto the wire, which would mean that I cannot rely on TCP as a reliable transport (which of course is the whole point of TCP).

This statement is incorrect. Just because you successfully write to a socket does not mean that data was received by the client or even left your machine. It just means it was successfully written to the internal TCP/IP buffers. TCP/IP will send that data in as many or as few blocks/frames as it deems fit. This is especailly true if the data you are sending is small, TCP/IP will delay to try to aggregate additional data into the frame.

If you are not seeing it in Wireshark, then you are likely not listening on the right interface, have a filter set, or a firewall of some sort is preventing you from sending the data. Take a look at any antivirous or other layere service providers in place to see who could be getting in there.

tcarvin
  • 10,715
  • 3
  • 31
  • 52
  • I see all of my other data on this interface, and in fact, there is only one interface available anyway. I think that the problem I am seeing must reside somewhere underneath my application, in the socket layer in Windows. As a follow-up, when you say that the send from .NET only goes down to the internal TCP/IP buffer, I have to wonder where Wireshark is performing the capture. Probably not at that level, but possibly when the network interface actually sends the data? If that is the case, then I don't really know what I would do to troubleshoot this. – Aron Sep 07 '11 at 20:10
  • Wireshark would be lower than the tcp/ip buffering...it would see the data after it was aggregated for example. Are you confident there is not any kind of anti-virus in place? Or filter active on your Wireshark capture? – tcarvin Sep 07 '11 at 20:52
  • 1
    When you say you see packets coming from the client and they are getting ACKed, are these on the same socket connection that you are not able to send on? – tcarvin Sep 07 '11 at 21:05
  • I'm confident that there is no AV in place on this machine, but it is a VM that I am running, if that matters. There is also no filter active on my Wireshark... I am actually seeing everything that I would expect to see, except occasionally when this error occurs. I don't think I mentioned it, but this is only happening once every 20-30 minutes. As for your last question, the packets coming from the client which are getting ACKed are on the same connection as the one that is failing me. – Aron Sep 07 '11 at 22:25
  • I'm not overly familair with additional issue caused by the VM in the mix. I do know that you can configure bandwidth limitations on VM instances? I regards to what is going on right when this probelm happens, are you seeing a black-out in just the data you send or, if you review the Wireshark log carefully, do you see any other outgoing ACKs that should have been sent by now but that are missing? I'm curious if the period in question is causing a complete outbound blackout on your connection or if it is just your data frame. – tcarvin Sep 08 '11 at 11:56
  • I am seeing other connections operating properly. There are other acks and data going in and out at the same time as this failed one. – Aron Sep 08 '11 at 18:34
  • It's a stumper. It's possible for Wireshark to drop some frames during a capture, but the connection is timing out as expected if your 4 bytes were sent but never ACKed, so there must be a real problem. Good luck, if I think of anything more I will post again. – tcarvin Sep 09 '11 at 12:08
0

If it's possible to disregard its limitations, would it be possible for you to rewrite the application to use TcpClient instead of Socket? If your app is bound to TCP/IP anyway, at least for this particular socket, you might as well use the abstraction -- it might help. Also, you could try to use the EndSend(IAsyncResult, SocketError) overload to see if the SocketError object contains anything useful.

If none of these software tweaks helps, I have to agree that the problem must be below the .NET software level, i.e. in firewall software (or hardware), antivirus, etc.

Asbjørn Ulsberg
  • 8,721
  • 3
  • 45
  • 61