Java thread dump: java.lang.Thread.State: WAITING (on object monitor)

Question

There are a large number of threads from our thread pool waiting for a connection indefinitely because our httpclient doesn't have any timeout.

Thread dump:

"pool-18-thread-400" #471 prio=5 os_prio=0 tid=0x00007fdf37a61000 nid=0x6ed7 in Object.wait() [0x00007fde8df9e000]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000007263acb18> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
    at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.doGetConnection(Unknown Source)
    - locked <0x00000007263acb18> (a org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool)
    at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager.getConnectionWithTimeout(Unknown Source)
    at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(Unknown Source)
    at org.apache.commons.httpclient.HttpClient.executeMethod(Unknown Source)

But we are calling future.cancel(true) with mayInterruptIfRunning flag set to true to kill such long-running threads after some time. These threads are still waiting for connections and are not getting freed up.

Question: Why are these threads not cleaned up by future.cancel? If future.cancel would not free up these threads, What are the alternative steps to kill such kinds of threads waiting for futile?

Adding more information about implementations

I can't share the exact code but providing some mock example

Our ThreadPoolexecutor is having Unbounded LinkedBlockingQueue and our future tasks are all callables and we are using executor.submit(callable) to execute our tasks.

public class MockThreadPoolExecutor extends ThreadPoolExecutor {
    public MockThreadPoolExecutor(int numThread) {
        super(numThread,numThread, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>());
        prestartAllCoreThreads();
    }

}

What is the code leading into this? Can you post it so we have a better idea of this, and where future.cancel is implemented for you. — OcelotcR, Dec 21 '17 at 14:42
Why don't you set timeout for your httpclient instead of killing mannualy ? One thread same topic : https://stackoverflow.com/questions/21445224/what-does-future-cancel-do-if-not-interrupting — Wilium, Dec 21 '17 at 14:53

score 1 · Answer 1 · answered Dec 21 '17 at 16:11

Your threads are waiting on synchronized (connectionPool) monitor in MultiThreadedHttpConnectionManager.doGetConnection which isn't responsible for interruption. According to the documentation of getConnectionWithTimeout increasing number of maxHostConnections and maxTotalConnections can help. It's also possible to specify timeout value in http.connection-manager.timeout which is 0 by default so threads are waiting for connection indefinitely.

/**
 * Gets a connection or waits if one is not available.  A connection is
 * available if one exists that is not being used or if fewer than
 * maxHostConnections have been created in the connectionPool, and fewer
 * than maxTotalConnections have been created in all connectionPools.
 *
 * @param hostConfiguration The host configuration specifying the connection
 *        details.
 * @param timeout the number of milliseconds to wait for a connection, 0 to
 * wait indefinitely
 *
 * @return HttpConnection an available connection
 *
 * @throws HttpException if a connection does not become available in
 * 'timeout' milliseconds
 * 
 * @since 3.0
 */

So, You are saying there is no way to kill or clean up the threads waiting for synchronized (connectionPool) if we missed setting out timeout? — Bhupesh Subramaniam, Dec 26 '17 at 09:06
Unfortunately yes. Such implementation looks pretty questionable to me, probably it makes sense to try other http client library (org.apache.httpcomponents for example) — Mikita Harbacheuski, Dec 26 '17 at 09:41

Java thread dump: java.lang.Thread.State: WAITING (on object monitor)

1 Answers1