2

I have a few web services on a Weblogic 10 server. Each of these is part of a larger system. Running locally and on our qa environment the system works flawless, replies fast, and as expected. Everything looks to be okay.

Before going into production we're going to stress test the system, thus see how much load we can have before reply time becomes to large. When testing the web services (e.g. using front end or SOAPUI) we hit a certain load (e.g. to many replies per sec or something like that, I'm not sure what exactly triggers the system to fail) we get the error listed below. I haven't got the slightest clue as to why. Seconds later the system replies flawless again, so I'm guessing that it has something to do with the number of requests...

Any ideas or hints is much appreciated! I'm lost here, so please - anything will help.

We're running: Weblogic 10.3.2, Spring 2.5.6 (for architectural reasons we cannot upgrade), Spring-WS 1.5.9 (for architectural reasons we cannot upgrade) and Stripes 1.5.4

<11-11-2011 08:43:58 CET> <Error> <HTTP> <BEA-101017> <[ServletContext@11242741[app:salesoverview-ws-web module:salesoverview-ws-web path:/salesoverview-ws-web spec-version:2.5], request: weblogic.servlet.internal.ServletRequestImpl@1fbbfc5[POST /salesoverview-ws-web/services HTTP/1.1 Accept-Encoding: gzip,deflate Content-Type: text/xml;charset=UTF-8 SOAPAction: "" User-Agent: Jakarta Commons-HttpClient/3.1 Content-Length: 425]] Root cause of ServletException.
org.springframework.ws.soap.saaj.SaajSoapMessageException: Could not write message to OutputStream: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error; nested exception is javax.xml.soap.SOAPException: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error
        at org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(SaajSoapMessage.java:169)
        at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:45)
        at org.springframework.ws.transport.support.WebServiceMessageReceiverObjectSupport.handleConnection(WebServiceMessageReceiverObjectSupport.java:97)
        at org.springframework.ws.transport.http.WebServiceMessageReceiverHandlerAdapter.handle(WebServiceMessageReceiverHandlerAdapter.java:57)
        at org.springframework.ws.transport.http.MessageDispatcherServlet.doService(MessageDispatcherServlet.java:230)
        Truncated. see log file for complete stacktrace

Caused By: javax.xml.soap.SOAPException: Error attempting to save SOAPPart. java.io.IOException: java.net.SocketException: Software caused connection abort: socket write error
        at weblogic.xml.saaj.SOAPMessageImpl.SOAPPart_writeTo(SOAPMessageImpl.java:1011)
        at weblogic.xml.saaj.SOAPMessageImpl.writeTo(SOAPMessageImpl.java:816)
        at org.springframework.ws.soap.saaj.Saaj13Implementation.writeTo(Saaj13Implementation.java:292)
        at org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(SaajSoapMessage.java:165)
        at org.springframework.ws.transport.AbstractWebServiceConnection.send(AbstractWebServiceConnection.java:45)
        Truncated. see log file for complete stacktrace
>

By digging BEA-101017 I found a little info about the from the Weblogic error dok - although this doesn't help me:

Error: [context] Root cause of ServletException.

Description: [context] Root cause of ServletException, which the Web application container caught while servicing the request.

Cause: The Web application container caught an unexpected exception.

Action: Check the exception for the exact error message.

sunlock
  • 212
  • 5
  • 17

2 Answers2

3

Assuming that the web service from your example doesn't access other web services (and therefore the above trace corresponds to your web service sending the response):

It seems that your web service, via SAAJ, is trying to write to a disconnected (or otherwise unavailable) socket. An usual cause for this is that the client has disconnected while waiting for the server reply.

I'd suggest to:

  1. Check if your client was waiting for too long before receiving the response, that could have caused it to disconnect.

  2. Check if the operating system might be having issues allocating sockets. Use 'netstat' or other monitoring tool (like TCPView on Windows) to check how many sockets are open (most operating system impose limits on the number of sockets allowed per user or globally).

  3. Ensure there are absolutely no network errors during your tests (shouldn't be the case if you are testing on localhost, but otherwise you need to ensure your network devices (routers, switches, other computers) are not dropping connections or packets. Perhaps this is happening when traffic load is high.

  4. Make sure you have no threading conflicts that could cause your web service to use or close other requests' sockets (this would be a rare situation especially if you are using Spring).

  5. Check this thread Official reasons for "Software caused connection abort: socket write error" and other possible causes of "Software caused connection abort" (note that the issue could be specific to your application server and operating system).

Hope that helps.

Community
  • 1
  • 1
jjmontes
  • 24,679
  • 4
  • 39
  • 51
  • 1. The client doesn't wait very long - matter of ms. 2. I'll have a look. 3. Shouldn't be anythnig here, all running on the same server. 4. I'm using Spring, but I'll have a look. 5. Already saw that, but I'm digging more. – sunlock Nov 11 '11 at 13:56
  • If you are running the load-testing tool on the same machine as the server, then I would try moving it. We always discourage generating load from the same machine you are testing - even when it doesn't cause errors like this, it doesn't result in realistic test data. – CMerrill Nov 11 '11 at 14:29
  • True, but the problem came when testing our qa environment where we have a dedicated machine for stress testing - I can reproduce the problem on my localhost. – sunlock Nov 11 '11 at 15:57
0

After debugging a lot I found out that the problem happened due to DB2 issues - we hit a corner of our database, which triggered an internal stack overflow, which then probagated to the Dao and onwards to the SOAP-part (only making it harder to detect due to Spring JDBC templates in the Dao).

A long story short and the issue was an uncaught exception, which by Spring-WS resulted in a "SaajSoapMessageException". The hint came from "Software caused connection abort: socket write error", but happened on the WS side (not client nor the communication between client/server).

Hint: Surround your database with try/catch and catch Exception thus being able to find the exact exception thrown. In my case it threw a DB2 exception ("SQLCode -1218") and this is normally used when you run out of resources (e.g. data source connections). I my case it was the SQL which DB2 didn't like - and really didn't like under load. I can't explain it, but it has to do with DB2s own internal resources - gah, go figure! :)

Thank you jjmontes, for hints and pointers, but it was not the problem in this case.

sunlock
  • 212
  • 5
  • 17