Very Bad WCF round trip time with more threads

Question

I have written a WCF application where client, server runs on the same machine. With few threads (<10) the average roundtrip time (client -> server -> client) is 400 milli sec. If I increase the threads to 200 then the average roundtrip time increases to 50 sec.

My WCF service is using basicHttp, Per-call, concurrent with 500 connections. Service is hosted in a console. Client is using the interface and communicates with service by opening a Channel using ChannelFactory. The data exchanged between server, client is at most 5 MB.

Server app.config

<system.serviceModel>
<behaviors>
  <serviceBehaviors>
    <behavior name="serviceBehavior1">
      <serviceMetadata httpGetEnabled="true" httpGetUrl="http://localhost:3999/TestWCFService/" httpsGetEnabled="false" httpsGetUrl="https://http://localhost:3999/TestWCFService/"/>
      <serviceDebug includeExceptionDetailInFaults="true" />
      <serviceThrottling maxConcurrentCalls="500" maxConcurrentSessions="500" maxConcurrentInstances="500" />
    </behavior>
  </serviceBehaviors>
</behaviors>
<bindings>
  <basicHttpBinding>
    <binding name="basicHttp" openTimeout="02:00:00" receiveTimeout="21:00:00" sendTimeout="21:00:00" maxBufferPoolSize="2147483647" maxBufferSize="2147483647" maxReceivedMessageSize="2147483647">
      <readerQuotas maxDepth="32" maxStringContentLength="2147483647" maxArrayLength="2147483647" />
    </binding>
  </basicHttpBinding>
</bindings>
<services>
  <service behaviorConfiguration="serviceBehavior1" name="TestWCF.TestWCFServer">
    <endpoint address="http://localhost:3999/TestWCFService/" binding="basicHttpBinding" bindingConfiguration="basicHttp"
      name="TestWCFServiceEndpoint" contract="TestWCF.ITestWCFInterface">
      <identity>  <dns value="localhost"/>  </identity>
    </endpoint>
    <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
    <host>
        <baseAddresses>  <add baseAddress="http://localhost:3999/TestWCFService/"/> </baseAddresses>
    </host>
  </service>
</services>

Server Code (The arguments Geometry, EntityDataFilter are defined in a separate DLL which is added as a reference in both client, server)

    [ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall, ConcurrencyMode = ConcurrencyMode.Multiple)]
internal class TestWCFServer : ITestServiceInterface
{
  public IList<EntityData> fetchEntityData(Geometry boundary, EntityDataFilter filter, string clientID)
  {
    // Fetch the EntityData from database inmemory cache with the given inputs.
    // I have a timer inside the whole method. This whole method is taking ~4 milli seconds.
  }
}

// Main class that starts the server
public static class TestWCFMain
{
  static void Main()
  {
    using (ServiceHost host = new ServiceHost(typeof(VSCacheService)))
    {
        host.Open();
        Log.Info("Server is UP !!!");
        Log.Info("<Press enter to shutdown server>");               

        Console.ReadLine();
    }
  }
}

Client app config

<system.serviceModel>
<bindings>
  <basicHttpBinding>
    <binding name="basicHttp" openTimeout="02:00:00" receiveTimeout="21:00:00" sendTimeout="21:00:00"
           maxBufferPoolSize="2147483647" maxBufferSize="2147483647" maxReceivedMessageSize="2147483647">
       <readerQuotas maxDepth="32" maxStringContentLength="2147483647" maxArrayLength="2147483647" />
    </binding>
  </basicHttpBinding>
</bindings>
<client>
  <endpoint address="http://localhost:3999/TestWCFService/" binding="basicHttpBinding"
      bindingConfiguration="basicHttp" contract="TestWCF.ITestWCFInterface" name="TestWCFServiceEndpoint" kind="" endpointConfiguration="" />
</client>

Client Code (The arguments Geometry, EntityDataFilter are defined in a separate DLL which is added as a reference in both client, server)

void fetchEntityDataFromServer(Geometry boundary, EntityDataFilter filter, string clientID)
{
    var channelFactor = new ChannelFactory<ITestWCFInterface>("TestWCFServiceEndpoint");
    channelFactor.Open();

    var proxy = channelFactor.CreateChannel();

    IList<EntityData> res = proxy.fetchEntityData(Geometry boundary, EntityDataFilter filter, string clientID);
}

I am stuck here. Any pointers to move forward will be a great help. Thanks in advance..

score 1 · Answer 1 · answered Mar 04 '16 at 23:37

I perform some tests with 200 threads with my WCF service. Client and service are placed on the same OS.

Client: 200 threads with 20 x call to service method

Service:

  [ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]

  <wsHttpBinding>
    <binding name="WSHttpBinding" sendTimeout="00:00:30" transactionFlow="false" maxReceivedMessageSize="2147483647">
      <security mode="None">
        <message clientCredentialType="None" establishSecurityContext="false" negotiateServiceCredential="false" />
        <transport clientCredentialType="None" />
      </security>
    </binding>
  </wsHttpBinding>

Results for different service method variants

Thread.Sleep(10)
- Average client response time: ~11ms
1MB response of random string
- Average response time: ~1.5min
- CPU usage: ~90-93%
1MB response of const string
- Average response time: ~450ms
- CPU usage: ~75-80%
5MB response of random string
- Average response time: ~6min
- CPU usage: ~95-99%
5MB response of const string
- Average response time: ~1,7min
- CPU usage: ~75-80%

Suspicions: Each call = 5MB response ? If yes then it sounds like a lot of data to serialize/transfer. In my opinion it's just hardware limitation.

Suggestions:

Do not open factory. channelFactory.Open()
Check channel factory cache option or cache it by yourself.
Try another bindings ...even theoretically slower wsHttpBinding with security mode="None"
If client and server are running on the same OS then try NetNamedPipeBinding.

score 0 · Answer 2 · edited May 23 '17 at 11:50

0

You should try increasing the number of concurrent connections from the client code that WCF sets up. i think threads are waiting until a connection can be setup thus increasing the average throughput.

See this post for and example: Multiple concurrent WCF calls from single client to Service

edited May 23 '17 at 11:50

Community

1
1

answered Mar 04 '16 at 09:34

robbie kouwenberg

42
3

Thanks for your reply. I have added System.Net.ServicePointManager.DefaultConnectionLimit = 500; Although my number of threads are 200. This has reduced the average round trip time but still it is very high.. The round trip time is around 8 seconds.. after some time (~2 min) it will come down to 500 milli seconds then after some time (~2-3 min) it will increase back to 8 seconds. Is there a way to reduce the round trip time to come down to 500 milli seconds. what is the best performance way to write the WCF service. I read netTCP is best but will it give a significant performance improvement? – Vijay Chukka Mar 04 '16 at 10:19
I think @adobrzyc is right in concluding that you are running into hardware limitations. In this case I would investigate scaling out to multiple machines. perhaps even give namedpipes a try – robbie kouwenberg Mar 09 '16 at 13:54

Very Bad WCF round trip time with more threads

2 Answers2