1

Every once in a while we get

Mar 31, 2015 7:24:32 PM org.apache.tomcat.util.threads.ThreadPool logFull
SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200) or check the servlet status

in the Catalina logs and server stops responding. The server is used very lightly and number of concurrent users not anywhere near 200 (2-3 on a busy day)

The thread dump shows 199 of these:

"TP-Processor200" daemon prio=10 tid=0x00002b513c31b000 nid=0x1c44 runnable [0x00002b514a9a7000]
java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
    - locked <0x00000007873208a0> (a java.io.BufferedInputStream)
    at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:628)
    at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:566)
    at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:693)
    at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:898)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
    at java.lang.Thread.run(Thread.java:662)

and one of

"TP-Processor4" daemon prio=10 tid=0x00002b513c21a000 nid=0x7470 in Object.wait() [0x00002b5135520000]
java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x0000000784a7a208> (a org.apache.tomcat.util.threads.ThreadPool)
    at java.lang.Object.wait(Object.java:485)
    at org.apache.tomcat.util.threads.ThreadPool.findControlRunnable(ThreadPool.java:339)
    - locked <0x0000000784a7a208> (a org.apache.tomcat.util.threads.ThreadPool)
    at org.apache.tomcat.util.threads.ThreadPool.runIt(ThreadPool.java:314)
    at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:676)
    at org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(ChannelSocket.java:879)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:690)
    at java.lang.Thread.run(Thread.java:662)

Can anybody figure out why it complaints about busy threads when they are clearly not?

H G
  • 11
  • 1
  • 2

1 Answers1

0

First, the InputStream method when used with sockets (i.e. SocketInputStream) - will block by default "until data is received, the end of the stream is detected, or an exception is thrown." A timeout value can be set with setSoTimeout to limit blocking on read to x ms, which when reached will throw a SocketTimeoutException.

A socket that is blocked reading in another thread can be shut down for input which should then signal to the reading thread to close the socket and exit.

This can be done by using the shutdownInput method on a socket:

//Shutting down input to socket...
socket.shutdownInput();

Then, you could still send to an output socket with:

//Write to socket again 
toSocket.print("is socket connection still available?\r\n"); 
//close socket 
socket.close();

Also depending on whatever "device" is being used to deliver the stream...i.e. if it's something like a container or a COM port this behavior may differ based on implementation or by vendor specification...if this is the case you might have a look at some of the suggestions offered here: Java InputStream blocking read - and as suggested possibly wrapping the input with DataInputStream to have things behave with more "file-like" consistency, i.e. ensuring that an EOFException is thrown if nothing is received.

Still another option might be to switch to a non-blocking connector implementation like NIO - where threads are not blocked during the time the connections are idle, and instead are monitored by special "poller" threads which serve to limit over-consumption of the "normal" threads in the thread-pool.

Some useful information on NIO and thread contention can also be found here: http://tutorials.jenkov.com/java-nio/server-socket-channel.html

rtreweek
  • 1
  • 1