WCF timing out too soon & 'Connection was closed unexpectedly' exception

Question

A WCF client returns the above exception after 120 seconds when calling a method which returns a string "success". Logs from the server show it executes fully and completes in around 3 minutes. With tracing enabled I see no errors thrown. Any help would be greatly appreciated as other suggestions from similar problems unfortunately haven't helped. Thanks!

EDIT - Solved: Turns out that the client was using a Smoothwall filtering device which was closing connections after two minutes.

Exception:

System.ServiceModel.CommunicationException The underlying connection was closed: The connection was closed unexpectedly.
Stack Trace:    at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)

InnerException:

System.Net.WebException The underlying connection was closed: The connection was closed unexpectedly.The underlying connection was closed: The connection was closed unexpectedly.
Stack Trace:    at System.Net.HttpWebRequest.GetResponse()
   at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)

As far as I can see, the bindings are set up correctly and the timeouts are beyond 120 seconds. Here's the client binding:

<bindings>
  <customBinding>
      <binding name="CustomBinding_IPHG" closeTimeout="00:05:00"
          openTimeout="00:05:00" receiveTimeout="00:15:00" sendTimeout="00:15:00">
          <security authenticationMode="SecureConversation" requireDerivedKeys="false">
              <localClientSettings maxClockSkew="00:30:00" />
              <localServiceSettings maxClockSkew="00:30:00" />
              <secureConversationBootstrap authenticationMode="UserNameOverTransport">
                  <localClientSettings maxClockSkew="00:30:00" />
                  <localServiceSettings maxClockSkew="00:30:00" />
              </secureConversationBootstrap>
          </security>
          <textMessageEncoding>
              <readerQuotas maxStringContentLength="5242880" />
          </textMessageEncoding>
          <httpsTransport maxReceivedMessageSize="6553600" useDefaultWebProxy="true" />
      </binding>
  </customBinding>
</bindings>
<behaviors>
<endpointBehaviors>
<behavior name="commonBehaviour">
  <dataContractSerializer maxItemsInObjectGraph="2147483647"/>
  <clientCredentials>
    <serviceCertificate>
      <authentication certificateValidationMode="PeerOrChainTrust"/>
    </serviceCertificate>
  </clientCredentials>
</behavior>
</endpointBehaviors>
</behaviors>

Here's the server binding:

<bindings>
  <customBinding>
    <binding name="CommonBinding" closeTimeout="00:05:00" openTimeout="00:05:00" receiveTimeout="00:15:00" sendTimeout="00:15:00">
      <transactionFlow />
      <security authenticationMode="SecureConversation" requireDerivedKeys="false">
        <secureConversationBootstrap authenticationMode="UserNameOverTransport">
          <localServiceSettings maxClockSkew="00:30:00" />
          <localClientSettings maxClockSkew="00:30:00" />
        </secureConversationBootstrap>
        <localServiceSettings maxClockSkew="00:30:00" />
        <localClientSettings maxClockSkew="00:30:00" />
      </security>
      <textMessageEncoding>
        <readerQuotas maxStringContentLength="41943040"/>
      </textMessageEncoding>
      <httpsTransport maxBufferSize="41943040" maxReceivedMessageSize="41943040" />
    </binding>
  </customBinding>
</bindings>
<behaviors>
  <serviceBehaviors>
    <behavior name="BehaviorRBS">
      <serviceMetadata httpsGetEnabled="true" httpGetEnabled="false" />
      <serviceDebug includeExceptionDetailInFaults="true" httpHelpPageEnabled="false" />
      <dataContractSerializer maxItemsInObjectGraph="2147483647" />
      <serviceCredentials>
        <userNameAuthentication userNamePasswordValidationMode="Custom" customUserNamePasswordValidatorType="PHGAPI.PHGUserNamePassValidator, PHGAPI"/>
        <serviceCertificate
                findValue="*.domainname.com"
                x509FindType="FindBySubjectName"
                storeLocation="LocalMachine"
                storeName="My" />
      </serviceCredentials>
      <serviceThrottling maxConcurrentCalls="200" maxConcurrentSessions="400" maxConcurrentInstances="600" />
    </behavior>
  </serviceBehaviors>
</behaviors>
<services>
    <service behaviorConfiguration="BehaviorRBS" name="PHGAPI.iAPI">
      <endpoint address="" binding="customBinding" bindingConfiguration="CommonBinding" contract="PHGAPI.IiAPI" />
    </service>
</services>

This is the interface for the method in question:

[OperationContract]
string SetSuspensions(List<Suspension> suspensions);

It looks like it was edited out, but did I read that you said the server completes its execution? This error is likely thrown from the server, not a timeout. To be clear, its the client response to a server error. — crthompson, Mar 19 '14 at 13:44
@paqogomez I've written to a log file at the server and it writes to the log immediately before the 'return "success";' line. This log entry happens after about 3 minutes which is roughly a minute after the client receives the exception. Because of this, I believe the server is executing fine. — Marcus, Mar 19 '14 at 14:03
@lavrik I notice other causes of this error could be due to serialisation, however it's a simple string being returned so I doubt it. — Marcus, Mar 19 '14 at 14:03
Can you post the relavent parts of your service interface, and the portion of web.config? — iamkrillin, Mar 24 '14 at 18:27
This is most likely related to the serialization of the result you are sending back. WCF has a great track record of providing inaccurate exceptions. — aevitas, Mar 25 '14 at 13:07
How the WCF service is hosted? What's the .NET version? Are you using the generated proxy on the client, or accessing channel factories directly? — noseratio, Mar 27 '14 at 08:54
@aevitas It just returns a string 'success' and the server is getting right down to the return statement. — Marcus, Mar 27 '14 at 10:05
@Noseratio it's hosted in IIS 7.5, .NET 4. Generated proxy on the client. On my machine it doesn't time out after 2 minutes, it seems isolated to this one customer's machine. Could it be something client side? — Marcus, Mar 27 '14 at 10:08
@IgorK, I'd first suggest looking at http://support.microsoft.com/kb/2538826 and implementing tweaks recommended there. BTW, the WCF runtime has been seriously improved for .NET 4.5. Other than that, have you checked if that customer's machine runs any antivirus/web filters or has a restrictive firewall/proxy config? — noseratio, Mar 27 '14 at 10:10
@Noseratio I will ask them to check their filters. This is a an overnight integration between a customer's database and ours and this particular method can take a few minutes to update quite a lot of data at our end. I've manually logged into a text file on the service and can see it's executing fully at our end. Tracing is showing up no errors either. — Marcus, Mar 27 '14 at 10:22
Two more things: [try disabling KeepAlive](http://stackoverflow.com/a/6535452/1768303), [check WCF and ASP.NET timeouts](http://blogs.msdn.com/b/hongmeig/archive/2010/03/06/timeouts-in-wcf-and-their-default-values.aspx). — noseratio, Mar 27 '14 at 10:31
if you have a proxy in the middle its possible that the timeout comes from that, depends on the proxy configuration. — Pedro.The.Kid, Mar 28 '14 at 15:35
@Pedro.The.Kid My current thinking is it's a proxy at the client that's causing this. They're investigating it at the moment. — Marcus, Mar 28 '14 at 15:41
@iamkrillin It's still not sorted. The customer has a Smoothwall proxy and we think that could be causing it with a persistent connection timeout. — Marcus, Apr 10 '14 at 08:16

score 4 · Answer 1 · answered Mar 24 '14 at 11:09

4

I would look into WCF and related configuration settings that have default values of 120 seconds. To that end, you may want to look into the HTTP.sys connectionTimeout value.

From the Microsoft documentation:
Connections can be considered inactive for the following reasons:

The HTTP.sys Timer_ConnectionIdle timer expired. The connection expired and remains idle.
The HTTP.sys Timer_EntityBody timer expired. The connection expired before the request entity body arrived. When it is clear that a request has an entity body, the HTTP API turns on the Timer_EntityBody timer. Initially, the limit of this timer is set to the connectionTimeout value. Each time another data indication is received on this request, the HTTP API resets the timer to give the connection more minutes as specified in the connectionTimeout attribute.
The HTTP.sys Timer_AppPool timer expired. The connection expired because a request waited too long in an application pool queue for a server application to dequeue and process it. This time-out duration is connectionTimeout.

The default value is 00:02:00 (two minutes).

http://www.iis.net/configreference/system.applicationhost/sites/sitedefaults/limits

Good luck.

answered Mar 24 '14 at 11:09

Seymour

7,043
12
44
51

Unfortunately this didn't work although I was really hopeful it would. The IIS site was adjusted to a 900 second connection time-out, however the same timeout happened on the client after 120 seconds. Is there a similar timeout on the client side (a console application)? – Marcus Mar 24 '14 at 15:00
The following may be helpful as the op seemed to have a similar issue: http://stackoverflow.com/questions/14266854/what-130-second-timeout-is-killig-my-wcf-streaming-service-call – Seymour Mar 24 '14 at 15:16
The log file located at C:\Windows\System32\LogFiles\HTTPERR\httperr1.log may provide some insight as to the cause of the timeout. – Seymour Mar 24 '14 at 15:19
In addition to the HttpSys log, the WCF trace file on the server and client would seem to be a good source of information, in particular the timing of any server communication/exception to the client. If you have not already setup tracing, the following link may help: http://msdn.microsoft.com/en-us/library/ms733025(v=vs.110).aspx – Seymour Mar 25 '14 at 11:28

score 2 · Answer 2 · answered Mar 23 '14 at 22:52

Tracing these issues can be a pain. Try the following to help track the problem down.

1) Identify where the problem is i.e. client or server. Start your Service in debug mode in the IDE and then use the WCF Test client rather than your own client to rule out client configuration problems.

2) Install and use Fiddler2 to inspect the HTTP messages going to\coming from the service. You will see if a response is coming back from the service and what's in it.

3) Assuming this is a synchronous call, holding a thread open for three minutes is bad for performance anyhow. Perhaps look into refactoring the service to either return the data a lot quicker or make it async .

1) Fiddler only works on the client stack. 2) even if the client done async the connection could still timeout — iamkrillin, Mar 30 '14 at 16:53

score 2 · Answer 3 · answered Mar 27 '14 at 12:42

Based on the information posted in your questions and the comments below, it sounds to me like its a timeout on the client section of this equation. Since the default timeout on the generated client is pretty small. Simply have your client modify their config to increase the timeout.

--exert (client config)--

<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
    </startup>
    <system.serviceModel>
        <bindings>
            <basicHttpBinding>
                <binding name="BasicHttpBinding_IService1" closeTimeout="00:10:00" openTimeout="00:10:00" receiveTimeout="00:10:00" sendTimeout="00:10:00"  />
            </basicHttpBinding>
        </bindings>
        <client>
            <endpoint address="http://localhost:8733/Design_Time_Addresses/WcfServiceLibrary1/Service1/"
                binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_IService1"
                contract="ServiceReference1.IService1" name="BasicHttpBinding_IService1" />
        </client>
    </system.serviceModel>
</configuration>

The bindings you are using are different, but all bindings have these properties. I usually increase all the timeouts, but the one thats going to be most important for you is the "receiveTimeout"

Thanks, I do think it's a client issue too. The client bindings already have 15 minute timeouts - this is listed in the original post. I've asked the customer to double check any web filters in case that is causing this issue. — Marcus, Mar 27 '14 at 17:03

score 2 · Answer 4 · answered Mar 27 '14 at 22:18

Where is the client located? (Same company network, same country)

what is the network speed at client side?

What is the size in bytes of the parameter List suspensions ?

Did you try to use Fiddler or some other network package software to capture the transmission?

In a prior job there was a case where every error in the server was logged to a different server; the issue was that an incoming message with 10,000 bad rows was sent to the log but not as a single failed error but as 10,000 times the same error.

Serializing this huge 200MB+ object to be sent to the other server was causing the internal network to be seen as 100% utilized and making the server to go 100%, thus IIS was not able to respond on time to any other request.

I'm not saying this may be your case, I'm just sharing a timeout situation I had where the problem was not in the method that timed out but in another component in the services.

Would you suggest using Fiddler on the client or server side? — Marcus, Mar 28 '14 at 15:20
Do you have access to both sides? if the problem is with a single client, then on the client side. If multiple clients, then check the traffic in the server — berXpert, Mar 28 '14 at 16:45

score -3 · Answer 5 · answered Mar 24 '14 at 10:38

Suggest you to use transaction scope in you business logic methods.

Still if you getting this kind of error, kindly stop your database server and restart. I think that might solve your problem. Because sometimes database traffic congestion takes more time than expected by the developer. So make less database calls from your application to server if your application is heavy.

WCF timing out too soon & 'Connection was closed unexpectedly' exception

5 Answers5