Client-Side CommunicationException while Service works properly

Question

Currently i am facing a problem i do not understand. I have an wcf client that calls a wcf service through several threads at the same time (both on the same machine). Sometimes, i encounter the well-known System.ServiceModel.CommunicationException

"An error occurred while receiving the HTTP response to xxx. This could be due to the service endpoint binding not using the HTTP protocol. This could also be due to an HTTP request context being aborted by the server (possibly due to the service shutting down). See server logs for more details." ,

sometimes it works. It seems completely random if the service call succeeds.

The request is very small, its just an (int, bool, enum) call. The request contains ca. 300-500 KB records from a MSSQL database.

I searched on hundreds on websites for a solution, increased timeout values, buffer values, request and response size values on the client and service side. I added [DataContract] and [DataMember] attributes were they were missing. And still there is no change.

I activated full tracing, just to see a very strange behaviour:

The calls from the client stop locally with the given exception - but the server processes them and sends a response, that never reaches the client. Check the time at the given trace - the client aborts, the server continues.

In the tracing, i see the heaviour like in this graph:

enter image description here

Please, can anyone help to stop this endless search?

Update 1:

I uploaded the client and server config, maybe this helps.

Update 2:

We included the config parameter for the ConnectionLimit (connection=80), as suggested from Yahia. Currently, it seems that this solved the problem, but we still try to reproduce the error.

Update 3:

Damn. After three hours, I can see the same behaviour again... We had another suggestion: As you can see, we are using quartz.net in the client, starting with 20 threads. The jobs the quartz engine executes connect to our service. Now I try to imagine what happens if, say 7 threads try to connect the service at the same time.

Update 4:

We have setup the tcp parameters in the registry as well as in the config. After a restart, experienced no change at all :(

These were the registry changes:

HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters - TcpNumConnections=65534
HKLM\System\CurrentControlSet\Services\Tcpip\Parameters - MaxUserPort=65534
HKCU\Software\Microsoft\Windows\CurrentVersion\Internet Settings - MaxConnectionsPer1_0Server=20
HKLM\Software\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_MAXCONNECTIONSPERSERVER - iexplore.exe=20
HKLM\Software\Microsoft\Internet Explorer\MAIN\FeatureControl\FEATURE_MAXCONNECTIONSPERSERVER - MyClientsExeName.exe=20

Update 5:

Client Trace Server Trace

Update 6:

[DataContract]
public class Currency
{
    [DataMember]
    public int Id { get; set; }

    [DataMember]
    public string Code { get; set; }

    [DataMember]
    public string Name { get; set; }

    [DataMember]
    public ForeignNoteDetails ForeignNoteDetails { get; set; }

    [DataMember]
    public CurrencyRates Rates { get; set; }

    [DataMember]
    public PreciousMetallDetails PreciousMetallDetails { get; set; }

    [DataMember]
    public CurrencyType Type { get; set; }
}

[DataContract]
public class ForeignNoteDetails
{
    [DataMember]
    public double CardholderBillingCurrencyCode { get; set; }

    [DataMember]
    public string Country { get; set; }

    [DataMember]
    public string CountryCode { get; set; }

    [DataMember]
    public int CurrencyUnit { get; set; }

    [DataMember]
    public List<string> Notes { get; set; }
}

[DataContract]
public class CurrencyRates
{
    [DataMember]
    public ExchangeRate PurchaseRate { get; set; }

    [DataMember]
    public ExchangeRate SellRate { get; set; }
}

[DataContract]
public class PreciousMetallDetails
{
    [DataMember]
    public string PreciousMetalType { get; set; }

    [DataMember]
    public double Fineness { get; set; }
}

Call to service:

        protected IEnumerable<Currency> GetCurrencyLevel(int id, bool netRate = true, RatesCalculationSource ratesSource = RatesCalculationSource.ReutersRates)
    {
        return this.calculationClient.GetCurrencyLevel(new RatesCalculationSetting() { CalculationLevelId = id, CalculateGrossRates = !netRate, Soruce = ratesSource });
    }

Client creation:

protected ICalculationServiceClientService calculationClient = IoC.DependencyManager.Resolve<ICalculationServiceClientService>();

Another call to the service (working):

this.calculationClient.DistributeTradingOfficeRatesLevels(branchOfficeLevelId, tradingLevelId);

Where this is defined as

void DistributeTradingOfficeRatesLevels(int branchOfficeRatesLevelId, int tradingOfficeRatesLevelId)

Update 7:

Exception details

I assume that you have already set the MaxReceivedMessageSize property on TransportBindingElement, both on server and client bindings. — Yiğit Yener, Mar 21 '13 at 10:42
Sadly, yes. The MaxReceivedMessageSize parameter is set to 6553600 on both the client and the service side in basicHttpBinding. — user2173257, Mar 21 '13 at 12:06
you need to change the setting globally (via registry)... Quartz.net might be doing something that prevents your config setting from being applied properly (perhaps they are reusing threads or similar)... — Yahia, Mar 22 '13 at 15:47
Ok, i'll try the combo of config AND registry. Does the setting for the internet explorer affect the whole system or do i have to enter the client's exe name as the registry key here? — user2173257, Mar 22 '13 at 15:55
please read the link I provided in my answer... it is a global setting for all "user-space applications" (except those using their own HTTP stack). — Yahia, Mar 22 '13 at 17:47
If your problem has not been resolved, yet, is it possible to see the trace from both the client side and the server side? — milanio, Apr 02 '13 at 12:58
Can you create a log of what is being sent exactly? (I am sure your problem is in the client side) — G.Y, Apr 04 '13 at 06:26
@G.Y: I'm pretty sure it's the client side. You mean logging the soap messages? — user2173257, Apr 04 '13 at 09:49
@milanio: I added pictures of the traces showing the specific situation: The server proceeds while the client errors out. — user2173257, Apr 04 '13 at 09:50
The method it fails on is related to currency right? can you please check that currency settings in client side and server side are same? (it's probably nothing but lets get it out of the way just to be extra sure) — G.Y, Apr 04 '13 at 10:03
@G.Y: I'm not sure what you mean. Currency is a class containing integer and double values. Which currency settings do you mean? (getting all out of the way is a favorite of mine :D) — user2173257, Apr 04 '13 at 10:08
@user2173257 1. do you get this exception with other classes as well? 2. do you import the types from your service or are they defined dually both in client and in server? 3. can you create a simple test service which will hold only this class and a client side which will use only this method to reproduce the problem without the rest of the code? — G.Y, Apr 04 '13 at 10:23
@G.Y: 1. Only when i call "GetCurrencyLevel" - but other methods use the currency class, too (but these methods don't fail) 2. The client holds a service reference, so the classes are defined only in the service. 3. This will take some time, because i fear it has to do with the whole huge construct — user2173257, Apr 04 '13 at 10:29
@user2173257 Alright, Can you show the code of this class "currency" and a code of another class in your project which don't fail - I think that something should be found by comparing the constructs and the syntax. — G.Y, Apr 04 '13 at 10:36
excellent - I will try to reproduce your problem and let you know in few minutes. — G.Y, Apr 04 '13 at 11:43
Can you split the object initializer from the call and test? protected IEnumerable GetCurrencyLevel(int id, bool netRate = true, RatesCalculationSource ratesSource = RatesCalculationSource.ReutersRates) { var myObj = new RatesCalculationSetting() { CalculationLevelId = id, CalculateGrossRates = !netRate, Soruce = ratesSource }; var result = this.calculationClient.GetCurrencyLevel(myObj); return result; } — G.Y, Apr 04 '13 at 11:55
In 30 years of programming trust that I seen problems that were solved by putting a lemon juice near the keyboard.. that were reproducible! so yes, when it comes to a non trivial bug - you sometime need to suspect even things like .. initializers came with .net4 ... WCF came in .net2 - maybe someone in MS didn't consider object initializers in imported services.. who knows ? :) just test it and tell me that we can clear this as well. — G.Y, Apr 04 '13 at 13:25
Can you show the details of the error message you get in wcfclient trace? I mean details of the row: "Throwing an exception". — milanio, Apr 04 '13 at 13:33
@G.Y: Yeah, i've seen strange things in the past, too. I wasn't sceptic - just wanted to know if you had something special in mind. I'll test this, but because we get the errors only after a few hours, i have to wait a day for the response. — user2173257, Apr 04 '13 at 13:48
Oh wait, stop... My bad... you getting the errors only after few hours.. so the method is being invoked successfully as it is.. damn - wrong path - the problem is not in the code. Let's try a different approach did you test it from several machines or just one machine? That start smelling like a faulty network card.. Your response please! — G.Y, Apr 04 '13 at 14:24
@G. Y: Hehe, a thought i had, too. The problem is that currently we don't have another machine to test. I could only move the client to another machine... (tested only on one machine so far) — user2173257, Apr 04 '13 at 14:27
@G.Y: Have you seen that client and service are on the same machine? — user2173257, Apr 04 '13 at 14:28
Ok, I'm ready to add my answer, please read my answer :) (you should have told me from the beginning this happens after few hours of work!) — G.Y, Apr 04 '13 at 14:32
@user2173257 even if both on same machine - network card is participant in the transfer loop... can you try 127.0.0.1 instead of the address you currently using - the error will not occur. — G.Y, Apr 04 '13 at 14:40
Is there anyway to delete all those comments, I would like them to think I got to this assumption just by watching your description lol — G.Y, Apr 04 '13 at 14:50

score 4 · Answer 1 · edited May 23 '17 at 12:25

4

HTTP protocol defines a limit of 2 connections to the same web server at the same...

Since you are using http the behaviour you see might be related to that limit...

There is a registry setting for this and a programatic way - you probably need to increase that limit on your client (depending on the quantity of threads you are using)...

Other relevant settings (client-side) can be found here.

edited May 23 '17 at 12:25

Community

1
1

answered Mar 21 '13 at 14:45

Yahia

69,653
9
115
144

Thanks for the answer. That's interesting - I'll check and post my results later. – user2173257 Mar 21 '13 at 15:00
This limitation does not explain why sometimes a request errors out while no other requests are running. Any ideas? – user2173257 Mar 21 '13 at 15:19
@user2173257 yes, it does... since http connections have usually some "keep-alive-time" (which is somewhere configurable but there is a default in the standard IIRC) it might still be an interference with any other previous connections between the client (any application!) and that specific server! – Yahia Mar 21 '13 at 16:06
See my update edit above. As long as this seems to solve the problem, i'll accept your great idea as the answer. Thank you! – user2173257 Mar 22 '13 at 08:55
@Yahia `HTTP protocol defines a limit of 2 connections to the same web server at the same`. can you point me to the section in the RFC that states that? – jgauffin Mar 28 '13 at 14:43
1

@jgauffin last sentence in [RFC 2616 Section 8.1.4](http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html) . – Yahia Mar 28 '13 at 21:06

G.Y · Accepted Answer · 2013-04-04T14:43:22.760

4

Ok, this is what happening - you have a faulty network card that is being overheated after few hours and when that happens it begins to short-circuit, as result your client side "thinks" that the server was shut down. End of story - 100% sure. where's my bounty?

btw: it is overheated either cause of bad design especially if you are using laptop or due to a damage already exists in it.

edited Apr 04 '13 at 14:43

answered Apr 04 '13 at 14:32

G.Y

6,042
2
37
54

I could not believe it in the first time, but it seems when changing the url to localhost, the network card gets ignored. After this change, we had no errors - now since days. GREAT, THANK YOU! – user2173257 Apr 09 '13 at 07:01

score 2 · Answer 3 · answered Mar 28 '13 at 21:17

I'd suggest to verify 2 points:

Throttling on the server, like

behavior name="Throttled"
serviceThrottling
maxConcurrentCalls="1"
maxConcurrentSessions="1"
maxConcurrentInstances="1"
Not closed WCF sessions - do you close all your sessions (client proxy) after used?
In my case when I enlarged all timeouts to max, I got situation when after X calls, service became "stuck". When I changed timeouts, I was received exception like yours.

It was several years ago, So i did not remember exactly. Also, I remember I found good article that was described how and why to change Service Instance mode to help this (sessions) issue.

Thanks for the answer. I'll try adding the trottling node to my behaviour. The answer to 2. is: no. Because we reuse them, we do not close a session after each call. — user2173257, Apr 02 '13 at 07:27
We've added (20 to be sure that all threads can act at the same time) to our behaviour, but again, no change — user2173257, Apr 04 '13 at 13:54

score 2 · Answer 4 · answered Apr 04 '13 at 06:00

Try to enable tracing at WCF side, you would get detailed information as to what is happening at server side.

Also sometimes WCF channel factory is closed abruptly when client sending big number of requests. we faced similar issue when we have used certificate authentication with WCF. In that we have deviced RetryPolicy around this transient error. if we get this error, we abort channel factory, recreate and send again rather than throwing exception.

 while (retryCount <= MAXRETRY)
            {
                try
                {
                    return _proxyService.GetData(); // _proxyService is WCF proxy class
                }
                catch (CommunicationException transientException)
                {
                     if (SLEEPINTERVAL> 0)
                    {
                        Thread.Sleep(SLEEPINTERVAL);
                    }
                    ((IClientChannel) _proxyService).Abort();
                    _proxyService = functionToCreateProxy();
                }

                retryCount++;
            }

As you can see, wcf is already enabled but showing strange behaviour. The recreating will be a "last chance"-way for me - i'm sure it's possible to do this without a workaround - there must be something wrong at my client side. — user2173257, Apr 04 '13 at 09:51

score 0 · Answer 5 · answered Oct 19 '15 at 00:44

0

I had the same problem. In my case it was my mistake. I closed the channelFactory (proxy) after starting threads.

answered Oct 19 '15 at 00:44

Totrov Oleg

11
1

Client-Side CommunicationException while Service works properly

5 Answers5

Linked