10

I have a fat VB.NET Winform client that is using the an old asmx style web service. Very often, when I perform query that takes a while or pass a large amt of data to a web service in a dataset, I get the subject error.

The error seems to occur in < 1 min, which is far less than the web service timeout value that I have set or the timeout value on the ADO Command object that is performing the query within the web server.

It seems to occur whenever I am performing a large query that expects to return a lot of rows or when I am sending up a large amount of data to the web service. For example, it just occurred when I was passing a large dataset to the web server:

System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
   at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.PooledStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at System.Net.Connection.SyncRead(HttpWebRequest request, Boolean userRetrievedStream, Boolean probeRead)
   --- End of inner exception stack trace ---
   at System.Web.Services.Protocols.WebClientProtocol.GetWebResponse(WebRequest request)
   at System.Web.Services.Protocols.HttpWebClientProtocol.GetWebResponse(WebRequest request)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at Smit.Pipeline.Bo.localhost.WsSR.SaveOptions(String emailId, DataSet dsNeighborhood, DataSet dsOption, DataSet dsTaskApplications, DataSet dsCcUsers, DataSet dsDistinctUsers, DataSet dsReferencedApplications) in C:\My\Code\Pipeline2\Smit.Pipeline.Bo\Web References\localhost\Reference.vb:line 944
   at Smit.Pipeline.Bo.Options.Save(TaskApplications updatedTaskApplications) in

I've been looking a tons of postings on this error and it is surprising at how varied the circumstances which cause this error are. I've tried messing with Wireshark, but I am clueless how to use it.

This application only has about 20 users at any one time and I am able to reproduce this error in the middle of the night when probably no one is using the app, so I don't think that the number of requests to the web server or to the database is high. I'm probably the only person using the app right now and I just got the error now. It seems to have to do everything with the amount of data being passed in either direction.

halfer
  • 19,824
  • 17
  • 99
  • 186
Chad
  • 23,658
  • 51
  • 191
  • 321

8 Answers8

3

Check your client's app.config binding setting and see if you have specified a Max message size. The default value for this is 65536

To change this, either put it in your binding configuration (maxReceivedMessageSize, maxBufferSize and maxArrayLength are key properties for this) in your app.config, or programmatically by changing the property on the binding, like so

System.ServiceModel.BasicHttpBinding binding = new System.ServiceModel.BasicHttpBinding();

// if you are getting a LOT of data back, you will need to up Message Size
binding.MaxReceivedMessageSize = int.MaxValue;

If this doesnt solve your issue, you will have to check the server side's event log for details. The message means that the client didnt expect the connection to close, but it did, and it could mean many different things.

If possible, use WCF for web services. It solves a lot of problems that ASMX services still suffer from.

To increase the server side transmission limit and execution timeout, use this

<configuration>
  <system.web> 
    <httpRuntime maxMessageLength="409600" executionTimeoutInSeconds="300"/> 
  </system.web>
</configuration>
brimble2010
  • 17,796
  • 7
  • 28
  • 45
Greg Olmstead
  • 1,551
  • 10
  • 22
  • It's not the size of the msg, tho that is a good guess. I know so because I have encountered the error in debug mode and then reset the execution down the same query path again and it works fine the 2nd time. The query executed should have been the exact same query. – Chad May 19 '10 at 05:01
  • As I try to capture more error info, I also see this error : Unable to read data from the transport connection – Chad May 19 '10 at 05:02
  • Strange, but it still may have something to do with the size of the message or the execution timeout. Is it possible that the message the first time could have been larger than the message the second time? Do you get any details with the "Unable to read data" message? Check your web.config and try including the code in my edit to increase the message size to 400MB and set a 5min timeout. – Greg Olmstead May 19 '10 at 13:30
  • Although I sometimes get an error and sometimes do not, I do not believe that the size of the data being returned had changed. I am 99.99% sure. I am running an app that is used by a very small group and I was testing at 3AM. The "Unable to read data" msg, I see now, was also in my orig error msg. If I read it correctly, that is not the underlying inner error that was first encountered, but instead the "forcibly closed" error was. I did make the web config changes you recommended and unfor, they did not help. Vote up of course, for useful suggestions, but still no solution. :-) – Chad May 19 '10 at 15:04
  • Is there anything in the server's event log? do you notice a consistent time interval between calling the service and it's failure? – Greg Olmstead May 19 '10 at 15:22
  • I looked in the event logs of the web server and the database server and saw nothing related. I was surprised not to find anything. It doesn't appear to be a consistent time interval, but instead random. – Chad May 19 '10 at 15:39
  • we need to find something that reliably reproduces this issue. Is there any scenario where this does not fail 100%? such as a query that only returns a smaller subset of data? Your Original Post makes it seem like you think its the volume of data being passed across the wire, however if that data is not changing, then it doesnt make sense to have anything to do with that – Greg Olmstead May 19 '10 at 17:25
  • Your question was "Is there any scenario where this does not fail 100%?" The answer is: It never fails 100% of the time. For operations that involve less data being passed, it works 100% of the time, for a "larger" volumes it fails occasionally, say 1 out of 3 times, though the operation appears to be identical each time it is expected. For very large amts of data, I think it would fail consistantly 100% of the time. I agree that it seems odd that if we were hitting an upper limit that it would fail 100% of the time as soon as the threshol;d was reached, yet there seems to be a clear trend. – Chad May 20 '10 at 04:40
  • when I say "does not fail 100% of the time" i mean is there any scenario where it works perfectly all the time, such as when the message size is cut in half, or something similar – Greg Olmstead May 20 '10 at 04:47
  • Yes, when the message size is very small, it works 100% of the time. – Chad May 24 '10 at 02:14
  • I'm placing a call with MS. They're doing network traces. Let's see what they have to say... – Chad May 24 '10 at 04:12
  • 3
    I'm revisiting a reallyoldpost. I thought I would provide an update from many years ago after talking to ms support. They told me that something (can't remember the hw term)in our network was dropping packets. That's all I remember, bottom line,a hardware/ network issue rather than a software/app issue. – Chad Jun 20 '14 at 00:20
  • I'm interested in how you solved this issue (I've got the same symptoms). Is there something you can do in the software like resending lost packets? I'm 90% sure that in my case the Web.config and App.config are alright regarding timeouts and message sizes, as well as the sqlcommand.timeout. EDIT: By any chance, could TOE/Chimney have been the issue, as described in this post? http://blogs.msdn.com/b/sql_protocols/archive/2008/04/08/understanding-connection-forcibly-closed-by-remote-host-errors-caused-by-toe-chimney.aspx. According to Symantec TOE/Chimney is known to lead to dropped packets. – Juergen Mar 05 '15 at 14:46
2

This code solved the issue for me:

Socket.ReceiveFrom(Buffer, Net.Sockets.SocketFlags.None, ReceiveEndpoint)
Socket.SendTo(Buffer, Length, Net.Sockets.SocketFlags.None, ReceiveEndpoint)

When I used the function with socketsflags the server/client didn't get disconnected again.

Pablo Claus
  • 5,886
  • 3
  • 29
  • 38
Frank Jansen of Lorkeers
  • 21
  • 1
2

I was having the exact same issue. Web service calls would fail with the same intermittent exception (~ once a day). It wasn't related to too large packet sizes or too small timeouts.

I ended up just putting in some retry logic which has worked around the problem. See: How can I improve this exception retry scenario?

Community
  • 1
  • 1
robbie
  • 1,103
  • 1
  • 10
  • 13
  • 1
    No, I don't yet have an answer. I called MS and they thought that it was a device whose max packet size was set to small. They needed a network person in my company to help them identify which device. It's been 6 mos and after constant nagging, they all but ignore me, refusing MS's help and not been of any help themselves. – Chad Nov 08 '10 at 14:37
1

If it is ASP.NET try setting the maxRequestLength to a higher value in the web.config file

<httpRuntime maxRequestLength="65536"/>

Do not increase the value too much as it might result in other errors like Access Denial etc. If it is a dataupload to a remote server, try slicing the data and sending in chunks.

Kind Regards,

Mafaz

sm2mafaz
  • 392
  • 3
  • 15
1

In my case, my generic HTTP handler (handler.ashx) was experiencing the unexpectedly dropped connection when trying to retrieve large files from a WCF service. In the end, the answer came in one Microsoft web page (https://msdn.microsoft.com/en-us/library/ms733742.aspx) and a call to Microsoft, who told me that one critical item on that page was misspelled (should have been "Streamed" instead of "Streaming"). The corrected code snippet from that page, applied to my WCF service's app.config file is this:

<system.serviceModel>
<bindings>
    ...
    <basicHttpBinding>
    <binding name="ExampleBinding" transferMode="Streamed"/>
        </basicHttpBinding>
    </bindings>
    ...
<system.serviceModel>

The key was to set the transfer mode.

Paul H. Harder II
  • 11
  • 2
0

For IIS 7 (in my case), the webservice's application pool's identity was "ApplicationPoolIdentity". I assigned to a local user and it worked just fine.

user007
  • 1,504
  • 2
  • 18
  • 51
0

Okay, i got same error. But in my case the problem was in AV software, which was blocking reporting services locally.

Andrew Kostenko
  • 270
  • 3
  • 5
0

try this ,it work for me for 10000 sent via network stream

defind SendBufferSize more than setting of the current TcpClient or optimize to your system if not define the value is 8192 that mean you can not send byte out more than this.

**YourInstantTcpClient**.SendBufferSize = 6550000

Sorry ,i'm thai people,may my english isn't good.

Ravindra Bagale
  • 17,226
  • 9
  • 43
  • 70
mezz
  • 1