4

I have a stuck thread in a Tomcat container, I would like to find the cause to avoid this problem.

Using jstack I have found it is stuck after a SOAP webservice call:

pool-2-thread-1" #27 prio=5 os_prio=0 tid=0x00007fd2315ce800 nid=0x39 runnable [0x00007fd2415d2000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    - locked <0x0000000087eb10e0> (a java.io.BufferedInputStream)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
    - locked <0x0000000087eb1138> (a sun.net.www.protocol.http.HttpURLConnection)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
    - locked <0x0000000087eb1138> (a sun.net.www.protocol.http.HttpURLConnection)
    at java.net.URL.openStream(URL.java:1045)
    at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.createReader(RuntimeWSDLParser.java:999)
    at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.resolveWSDL(RuntimeWSDLParser.java:400)
    at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:231)
    at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:209)
    at com.sun.xml.ws.wsdl.parser.RuntimeWSDLParser.parse(RuntimeWSDLParser.java:178)
    at com.sun.xml.ws.client.WSServiceDelegate.parseWSDL(WSServiceDelegate.java:364)
    at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:322)
    at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:231)
    at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:212)
    at com.sun.xml.ws.client.WSServiceDelegate.<init>(WSServiceDelegate.java:208)
    at com.sun.xml.ws.spi.ProviderImpl.createServiceDelegate(ProviderImpl.java:119)
    at javax.xml.ws.Service.<init>(Service.java:77)

Which timeout is missing in the configuration? I'm using the standard options.

Tobia
  • 9,165
  • 28
  • 114
  • 219
  • Could you find out what the problem is? I'm stuck at a similar problem... – markus Mar 31 '22 at 18:50
  • 1
    I didn't find the root cause of the infinite timeout. But I found that by default Spring Scheduled annotation uses a single-threaded pool, so, if it get stuck all scheduled jobs will stop. Then I suggest to use a multi-threaded pool setting. – Tobia Apr 01 '22 at 08:29
  • Did you try to set sun.net.client.defaultConnectTimeout and sun.net.client.defaultReadTimeout values when starting the JVM? ex : -Dsun.net.client.defaultConnectTimeout="1000" (one second). See https://docs.oracle.com/javase/7/docs/technotes/guides/net/properties.html – pringi Apr 01 '22 at 10:17
  • @pringi is it possible to set them runtime? I can change the code but I cannot easily change the startup script of my application. Thanks – Tobia Apr 03 '22 at 13:13
  • Could it be that you open to many connections without reusing / closing them? – fsch Apr 07 '22 at 08:05
  • @pringi Thanks for your responses. Apparently in our case the problem was not on the consumer but on the producer side. Due to a particular nasty producer side side-effect the producer all of a sudden suffered from heavily increased response times which led in turn to all consumer threads being stuck at java.net.SocketInputStream.socketRead0 Thanks for your inputs! – markus Apr 11 '22 at 09:23

1 Answers1

2

Your client tries to download the WDSL file of the SOAP service from a remote URL. HttpURLConnections have by default an infinite socket read timeout and an infinite connect timeout. In your case the connection seems to be stuck, while reading from the server. Since the read timeout is infinite, you are stuck forever or at least until the connection is reset (e.g. by a router or the server).

As already mentioned in one of the comments, the default timeouts can be configured through the system properties sun.net.client.defaultReadTimeout and sun.net.client.defaultConnectTimeout (as documented here). You could set these properties also at runtime, but you would have to do this before the properties are read. Unfortunately, the properties are read only once when the internal NetworkClient class is loaded. I haven't tried it out, but it is very unlikely that you will be able to set the properties within a Tomcat container, before this class gets loaded. If you should have a chance to change the start script of the Tomcat, I would heavily recommend to set explicit timeouts (especially in case of a production setup).

Setting explicit timeouts, is in general a good approach. But timeouts will only solve the described symptom, but not your actual problem: Your SOAP client relies on the successful download of a WSDL file from a remote location. Instead of downloading the WSDL file from a remote location, you could load it from a local resource. There are two possible ways to do this:

A) If you should have access to the (generated) client code, you could replace the remote URL of the WSDL location with the URL of a resource file on your classpath. See this answer for an example.

B) If manual adaptation of the client code isn't an option, you can use a catalog file, as documented here.

rmunge
  • 3,653
  • 5
  • 19