4

I am facing a strange situation where I have a thread pool with maxPoolSize as 100 and queue size as 30 and corePoolSize as 50. After application is restarted in some time I can see RejectedExecution errors in logs which state that even though threads are available , task was rejected.

org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor@593a81ea[Running, pool size = 100, active threads = 3, queued tasks = 17, completed tasks = 57463]]

I took thread dump many times and found that threads of this pool are doing Timed_Waiting. Above log is correct that only 3 are runnable as per dump also but I am not able to understand why rest of the 97 threads are not available to the system. Below is the thread dump which is common for 97 threads : 

YYYThreadPoolExecutor-100" #6414 prio=5 os_prio=0 tid=0x00007f0830dce800 nid=0x3a39 waiting on condition [0x00007f064b6b5000]
   java.lang.Thread.State: TIMED_WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007f0a32b21088> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)

System is a 16-core box with 100 GB RAM. Current cpu utilisation is just 103% for 16 cores and RAM utilisation is <15GB.

Even if I increase thread pool size to 500 also , same situation arrives with a delay.

Has anybody faced this situation before? Thanks in advance.

Vineet Kasat
  • 994
  • 7
  • 14

1 Answers1

4

Refer to oracle documentation about ThreadPoolExecutor 

public ThreadPoolExecutor(int corePoolSize,
                          int maximumPoolSize,
                          long keepAliveTime,
                          TimeUnit unit,
                          BlockingQueue<Runnable> workQueue,
                          RejectedExecutionHandler handler)

Creates a new ThreadPoolExecutor with the given initial parameters and default thread factory.

Parameters:

corePoolSize - the number of threads to keep in the pool, even if they are idle, unless allowCoreThreadTimeOut is set

maximumPoolSize - the maximum number of threads to allow in the pool

keepAliveTime - when the number of threads is greater than the core, this is the maximum time that excess idle threads will wait for new tasks before terminating.

unit - the time unit for the keepAliveTime argument

workQueue - the queue to use for holding tasks before they are executed. This queue will hold only the Runnable tasks submitted by the execute method.

handler - the handler to use when execution is blocked because the thread bounds and queue capacities are reached

From the details you have provided, you have not reached Thread pool bounds due to huge number but you might have reached queue capacity.

You don't need high number for number of threads. Instead you can increase queue capacity.

Generally queue size will be configured higher value to number thread pool size. maximumPoolSize can be configured to number of CPU cores in your machine : Runtime.getRuntime().availableProcessors()

Ravindra babu
  • 37,698
  • 11
  • 250
  • 211
  • Hi When server restarts I can see 100 threads being reached, i.e. upper bound touched and active threads = 100 ,but in few seconds , active threads drops suddenly to 5 and I get tasks getting rejected. I took thread dumps but those threads are just in TIMED_WAITING state, not locked for any monitor/IO. I cant increase queue size beyond a limit as it will slow down my system as hits are coming at a decent rate. Do u have any idea where those threads might be which are in system but not available to system – Vineet Kasat May 12 '16 at 12:16
  • Can you post your code to get better understanding? - ThreadPoolExecutor & Runnable/Callable part is good enough – Ravindra babu May 12 '16 at 12:22
  • I cant paste the whole code, but I can give you a brief if that helps. I have a threadpoolexecutor where tasks are submitted. In those tasks, another threadpoolexecutor is invoked and a countdown latche is present at both levels. I have taken care of using finally to countdown the latche in case of any unhandled exceptions. Await on latche is time driven. Inner executor gets 8-9 hits from parent task and this is the ine getting rejected task. Parent is accepting tasks normally. – Vineet Kasat May 12 '16 at 12:34
  • Can you post Inner executor constructor code? I just want to see the parameters – Ravindra babu May 12 '16 at 12:42
  • outer.executor.thread.pool.size=50 outer.executor.thread.max.pool.size=100 outer.executor.thread.keepAliveSeconds=1800 outer.executor.thread.queue.capacity=30 inner.executor.thread.pool.size=50 inner.executor.thread.max.pool.size=100 inner.executor.thread.keepAliveSeconds=1800 inner.executor.thread.queue.capacity=30 – Vineet Kasat May 12 '16 at 14:07
  • 1
    If you queue capacity is 30, you don't need even 30 threads. Definitely you are reaching queue capacity and hence getting rejected execution handler. What are you constraints in increasing the capacity of worker queue? – Ravindra babu May 12 '16 at 14:25
  • I agree that my queue size is less. But why is active threads not touching even core pool size. There is negligible cpu/memory load – Vineet Kasat May 12 '16 at 14:44
  • If your queue size is 30, you won' t need core pool size of 50 threads. – Ravindra babu May 12 '16 at 15:45
  • 1
    Have a look at source code of http://grepcode.com/file_/repository.grepcode.com/java/root/jdk/openjdk/6-b14/java/util/concurrent/ThreadPoolExecutor.java/?v=source execute method – Ravindra babu May 12 '16 at 15:54
  • Thanks I had looked at that code previously as well but this time with a new perspective and I know the reason for my issue now. Thanks – Vineet Kasat May 13 '16 at 07:46