4

We're using ServiceStack 3.9.71.0 and we're currently experiencing unexplained latency issues with clients over a WAN connection.

A reply with a very small payload (<100 bytes) is received after 200ms+.

The round-trip-time (RTT) on the link is about 40ms due to the geographical distance. This has been verified by pinging the other host and using a simple echo service to test the latency of a TCP connection.

Both ping and echo test show latencies which are in line with expectations. Getting a reply from our ServiceStack host takes much longer than expected.

We've verified that:

  • WAN link is only running at 25% of capacity (no congestion)
  • No QOS is employed on the WAN link
  • same host gives fast reply to same request from a different host on local network
  • delay is not caused by our code processing the request

We've now stumbled across Nagle's algorithm and that it can mean delays for small requests on WAN networks (http://blogs.msdn.com/b/windowsazurestorage/archive/2010/06/25/nagle-s-algorithm-is-not-friendly-towards-small-requests.aspx).

In .NET it can be disabled by setting TcpClient.NoDelay = true (https://msdn.microsoft.com/en-us/en-US/library/system.net.sockets.tcpclient.nodelay(v=vs.110).aspx).

How can this be disabled for ServiceStack's TCP handling?

EDIT: I don't think that this is a duplicate of HttpWebRequest is slow with chunked data. The mentioned question covers HttpWebRequest which isn't used by ServiceStack. ServiceStack uses HttpListener which also happens to be controlled / managed by the mentioned ServicePointManager. We're going to conduct a test to see whether setting ServicePointManager.UseNagleAlgorithm = false solves the issue.

Community
  • 1
  • 1
Timo Kosig
  • 845
  • 10
  • 23
  • Incase you didnt find any such way to disable nagle's alogrithm then , i suggest downloading the open source code of ServiceStack and search for Socket in the all the files , add an option to all such functions which create sockets to allow TCP_NODELAY option . add code to source to set TCP_NODELAY using setsocketopt function . https://msdn.microsoft.com/en-us/library/windows/desktop/ms740476(v=vs.85).aspx. After you might as well commit that changes on open source repo. This particular example explains why high level API's hamper low level tuning. – Vikram Bhat Apr 28 '15 at 11:14
  • Hi Vikram, thanks for your suggestion. ServiceStack has their source code for v3 publicly hosted on Github here: https://github.com/ServiceStack/ServiceStack/tree/v3. I've done a quick search but I don't think they use `Socket` directly. It looks like they're using `HttpListener` - any idea if TCP_NODELAY can be set for an `HttpListener`? – Timo Kosig Apr 28 '15 at 11:25
  • hi Timo, I am currently working on linux but I did search the repo and found out that the API where you need to add changes are in /src/ServiceStack.Server/Messaging/Rcon/Client.cs & /src/ServiceStack.Server/Messaging/Rcon/Server.cs . the sockets are create in functions connect(Client.cs) and start(Server.cs). Good luck, will make changes if necessary when I have time. – Vikram Bhat Apr 28 '15 at 11:36
  • Vikram, not to worry about those two files - AFAIK they're part of a messaging queue implementation which we don't use in our project so they shouldn't really have an impact if I'm not mistaken. – Timo Kosig Apr 28 '15 at 11:38
  • 1
    possible duplicate of [HttpWebRequest is slow with chunked data](http://stackoverflow.com/questions/18465504/httpwebrequest-is-slow-with-chunked-data). Since the library uses `HttpListener`, you may be able to control this using `ServicePointManager`. Alternatively, while not ideal, you could force the issue by sending extra data. Or, you could just live with the delay (your post is not clear why a 200ms delay is actually a problem...normally it wouldn't be). – Peter Duniho Apr 28 '15 at 17:10
  • @PeterDuniho sending extra bytes , precisely a payload of 64k will not help that much unless the packets are too infrequent and bandwidth is very large, larger packet takes longer time to reach due to transmission delays , moreover there is more chance of packet loss due to errors for large packets . Disabling the naggle algorithm is only reliable solution. – Vikram Bhat Apr 29 '15 at 07:42

1 Answers1

2

I think you provided an answer in your Update UseNagleAlgorithm = false should solve this issue. But be careful because ServicePointManager.UseNagleAlgorithm = false; is a global settings which means it will turn off this algorithm for all of your endpoint and for all of your requests in the entire App Domain. When you call more than one service endpoints (usually that is the case) with mixed sized of Request it will bite back. So you should consider setting this only for one specific ServicePoint, you can acquire it by:

ServicePoint sp = ServicePointManager.FindServicePoint(<uri>);
sp.UseNagleAlgorithm = false;

and not set it globally

Here is an article about it: https://technet2.github.io/Wiki/blogs/windowsazurestorage/nagles-algorithm-is-not-friendly-towards-small-requests.html

Dinesh
  • 3,652
  • 2
  • 25
  • 35
Major
  • 5,948
  • 2
  • 45
  • 60