0

We have a WCF service hosted in a windows service which hangs after long periods of inactivity: ie after the weekend.

This behavior happens at different locations.

The service uses WSHttpBinding which is set to use transport security a Custom serviceAuthorization authorizationPolicy which uses windows authentication.

The Spring framework is used. Spring.ServiceModel.Activation.ServiceHostFactory creates the Service hosts.

Service throttle is set to 200 sessions, instances and calls. There are at most 15 users of the system.

Tracing is enabled and set to Warning which should let me know about throttle issues. There are no messages in the service trace log. There are no messages in the Event log which look relevant. There are no relevant logs in HTTPPerf logs. We log considerably in the server side application but there is no activity being recorded when the system hangs. It is a total black box when the system hangs.

The client fails with the following message. 08:13:32.014 [1] ERROR App - System.TimeoutException: Client is unable to finish the security negotiation within the configured timeout (00:00:59.9941374). The current negotiation leg is 1 (00:00:59.9863206). ---> System.TimeoutException: The request channel timed out while waiting for a reply after 00:00:59.9628702. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'http://localhost:8080/OrderManagementService.svc' has exceeded the allotted timeout of 00:00:59.9690000. The time allotted to this operation may have been a portion of a longer timeout. ---> System.Net.WebException: The operation has timed out at System.Net.HttpWebRequest.GetResponse()

I have spent many hours searching for relevant information on this.

I do not think this is related to inactivity timeout as this should be recorded in the trace log.

Only thing that I can think of is related to Active Directory credential caching or something of that nature or that it is related to the use of the Spring framework.

Any help would be greatly, greatly appreciated.

Am thinking of moving away from WSHttpBinding or WCF altogether as this is an unacceptable situation.

Service side configuration is as follows.

<system.serviceModel>
<bindings>
  <wsHttpBinding>
    <binding name="WSHttpBinding" closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00" bypassProxyOnLocal="false" transactionFlow="false" hostNameComparisonMode="StrongWildcard" messageEncoding="Text" textEncoding="utf-8" useDefaultWebProxy="true" allowCookies="false" maxBufferPoolSize="2147483646" maxReceivedMessageSize="2147483646">
      <readerQuotas maxDepth="2147483646" maxStringContentLength="2147483646" maxArrayLength="2147483646" maxBytesPerRead="2147483646" maxNameTableCharCount="2147483646" />
      <reliableSession ordered="true" inactivityTimeout="00:10:00" enabled="false" />
      <security>
        <transport></transport>
      </security>
    </binding>
  </wsHttpBinding>
</bindings>
<behaviors>
  <serviceBehaviors>
    <behavior name="ServiceBehavior">
      <serviceMetadata httpGetEnabled="true" />
      <serviceDebug includeExceptionDetailInFaults="true" />
      <serviceAuthorization principalPermissionMode="Custom">
        <authorizationPolicies>
          <add policyType="Kodi.Kodiak.Security.AuthorizationPolicy, Kodi.Kodiak.Security" />
        </authorizationPolicies>
      </serviceAuthorization>
      <serviceCredentials>
        <windowsAuthentication includeWindowsGroups="true" allowAnonymousLogons="false" />
      </serviceCredentials>
      <serviceThrottling maxConcurrentCalls="200" maxConcurrentSessions="200" maxConcurrentInstances="200" />
      <dataContractSerializer maxItemsInObjectGraph="2147483646" />
    </behavior>
  </serviceBehaviors>
  <endpointBehaviors>
    <behavior name="rest">
      <webHttp />
    </behavior>
  </endpointBehaviors>
</behaviors>
<services>
  <service behaviorConfiguration="ServiceBehavior" name="OrderManagementService">
    <endpoint address="" binding="wsHttpBinding" bindingConfiguration="WSHttpBinding" contract="Kodi.Kodiak.Services.ServiceContracts.IOrderManagementService" />
    <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
  </service>
</services>

SPRING CONFIG

<object id="OrderManagementService"
    singleton="false"
    type="Kodi.Kodiak.Services.OrderManagementService, Kodi.Kodiak.Services"
    scope="session">
 </object>
tomasat
  • 578
  • 8
  • 11
  • does it works after restart? can you host this app on IIS to see how it works there? – Brijesh Mishra Jul 03 '12 at 12:08
  • how are you 'fixing' this after the weekend? restarting the windows service? is the wcf endpoint reachable (eg by telnet test) when the client tries to connect monday morning? (try this before your client does) – wal Jul 03 '12 at 12:47
  • Till you unable to find solution , keep service Alive like this. http://stackoverflow.com/questions/3466490/how-to-set-keep-alive-interval-for-http-connection-in-wcf – Vivek Jul 03 '12 at 13:05
  • Brijesh, Yes it works after the restart. Can not host in IIS because the windows service is connecting to other systems and needs to maintain state, connections etc. Wal, we restart the windows service when we run into this issue. By the way, this does not happen every monday so this has been a little hard to pinpoint. – tomasat Jul 04 '12 at 13:52
  • im curious, what does enabled="false" do in ReliableSession configuration? – Zee Jul 12 '12 at 11:39
  • Did you try to call the service second time after getting an error? Are subsequent calls return error as well? – Dmitry Harnitski Jul 12 '12 at 16:40
  • Regarding the reliableSession enabled false issue. http://stackoverflow.com/questions/1241568/wcf-is-inactivitytimeout-effective-when-reliablesession-enabled-false – tomasat Jul 12 '12 at 22:50
  • Dmitry, yes subsequent calls also return errors. The operations just simply stop responding. No logs, nothing. – tomasat Jul 12 '12 at 22:51
  • You need somehow to get server side error details. Unfortunately client error provides no meaningful info. Do you see any errors in event log? Add custom logging and check if you can catch exception. – Dmitry Harnitski Jul 13 '12 at 02:29
  • Our final solution consisted of creating a scheduled task which runs a batch file which restarted the windows service once a day. This is pretty much a brute force solution but we have not had any problems since. I guess there is a good reason reason why IIS recycles worker processes once a day by default. – tomasat Nov 19 '12 at 15:24

2 Answers2

0

How the clients are calling the service? If they are calling the service through the SvcUtil generated proxy; are they closing the proxy connection properly? I think the clients are not closing the connections or simply the connections are not getting closed properly.

One important thing is, you should avoid using using statement while creating proxies.

A better approach would be something like this,

ServiceClient client = null;

try
{
  client = new ServiceClient();
  client.CallMethod();
}
finally
{
  client.CloseConnection(); // extension method
}

public static void CloseConnection(this ICommunicationObject client)
{
   if (client.State != CommunicationState.Opened)
   {
      return;
   }

   try
   {
      client.Close();
   } 
   catch (CommunicationException)
   {
      client.Abort();
      throw;
   }
   catch (TimeoutException)
   {
      client.Abort();
      throw;
   }
   catch (Exception)
   {
      client.Abort();
      throw;
   }
}
VJAI
  • 32,167
  • 23
  • 102
  • 164
  • Hello Mark, yes we are calling the service in a similar way. We do not use the using statement. We use a ChannelFactory to create a client and use try.. finally in a similar fashion as you posted. – tomasat Jul 09 '12 at 21:30
0

I think this can be done by maintaining the client and server Binding End Points and Behavior same, the timeout values specified in the Client and Server Configs Should be Same.

Abdur Rahman
  • 11
  • 2