Java ExecutorService - threads in waiting state

Question

Use Case: Create a new thread every time I need to process a job.

Current Implementation: I am using Executor Service with fixed size thread pool, of say 50. For every job, I submit a new thread to executor service.

Issue: Once the job is completed, the threads do not die and go into waiting state. (waiting at sun.misc.unsafe.park)

Analysis: As per this link (WAITING at sun.misc.Unsafe.park(Native Method)) and other sources on the net, this is a valid scenario, and the threads go into waiting state, waiting for some task to be given to them.

Question: From Java mission control, I am able to deduce that the threads are not using any resources and are not in dead lock. So that is good. But consider a time frame when lots of jobs are submitted and all 50 threads in the pool were instantiated. And after that all the 50 threads are going to be alive even though the job submission rate might have decreased. I cannot shut down the executor service also, since it needs to be alive forever waiting for jobs to be submitted. If I create, normal threads, I see that the threads die after their job is done. But in that case there is no tab in the max amount of threads being created. So in a peak time, we may run into a situation where more threads are created than what the JVM can handle.

How can this scenario be handled in the best possible way. Should we ignore the threads being in the waiting state or should I go for any other implementation.

The behavior that I am trying to implement is more like autoscaling. Span more servers(threads in this case)during peak time. And terminate the extra servers and maintain a minimum server count when the load is not that high.

Answer 1

Use the ThreadPoolExecutor and set its keepAliveTime attribute via any one of its constructors.

Answer 2

It can be done using ThreadPoolExecutor. However it won't do what you expect it to do. Following constructor can be used to create a ThreadPoolExecutor.

ThreadPoolExecutor(int corePoolSize, int maximumPoolSize, long keepAliveTime, TimeUnit unit, BlockingQueue workQueue)

Let me break down it's behavior as documented. When a task is submitted

1. If poolSize is less than corePoolSize, a new thread is created, even if there are idle threads.
2. If poolSize is equal to the corePoolSize then task is added to the queue. It won't create new threads until queue is exhausted.
3. if workQueue is exhausted then new thread is created till poolSize becomes maximumPoolSize.
4. If poolSize is equal to the maximumPoolSize throw RejectedExecutionException

So now suppose we set core size to 5, max size to 10 and submit 100 tasks. Nothing will happen if we created our pool object with the Executors class. As pools created by it use LinkedBlockingQueue, with default constructor, which sets the queue capacity to measly Integer.MAX_VALUE (2147483647).

Following is the code from Executors

public static ExecutorService newFixedThreadPool(int nThreads) {
    return new ThreadPoolExecutor(nThreads, nThreads,
                                  0L, TimeUnit.MILLISECONDS,
                                  new LinkedBlockingQueue<Runnable>());
}

Default constructor in LinkedBlockingQueue

public LinkedBlockingQueue() {
    this(Integer.MAX_VALUE);
}
public LinkedBlockingQueue(int capacity) {
...

Option to create ThreadPoolExecutor directly still remains, however that's not much helpful. Lets examine that. Suppose we create ThreadPoolExecutor object using following code.

ArrayBlockingQueue<Runnable> workQueue = new ArrayBlockingQueue<>(MAX_QUEUE_SIZE);
ThreadPoolExecutor poolExecutor = new ThreadPoolExecutor(CORE_POOL_SIZE, MAX_POOL_SIZE, IDLE_LIFE_TIME, TimeUnit.SECONDS, workQueue);

Where MAX_QUEUE_SIZE is 10. Number of maximum task that can be submitted can be found by following formula.

MAX_TASKS = MAX_POOL_SIZE + WORK_QUEUE_CAPACITY

So if max pool size is 10 and work queue size is also 10, then 21st task will be rejected, if no threads are free.

It's important to keep in mind that it doesn't give us desired behavior. As threads are only killed if there are more threads than corePoolSize. Thread pool increases more than corePoolSize only if workQueue has been exhausted.

So maxPoolSize is a failsafe option to avoid queue exhaustion. Not the other way around. Max pool size is not intended to kill idle threads.

If we set queue size too small, we risk task rejection. If we set it too high, poolSize will never cross corePoolSize.

May be you can explore ThreadPoolExecutor.setRejectedExecutionHandler. And save rejected task a separate queue which will send tasks to workQueue once workQueue.capacity becomes less than the max capacity periodically. But that seems a lot of work without equivalent gain.

Answer 3

And after that all the 50 threads are going to be alive even though the job submission rate might have decreased. I cannot shut down the executor service also, since it needs to be alive forever waiting for jobs to be submitted.

...

How can this scenario be handled in the best possible way. Should we ignore the threads being in the waiting state or should I go for any other implementation.

I think the answer is, you should ignore them. Threads are pretty darn efficient these days and certainly 50 dormant threads aren't going to affect the runtime of your application in any way. It would be different if you are talking about a much large number of threads or a whole series of different thread pools all hanging around.

That said, as mentioned, if you want your threads to timeout then you will need to specify a different "core" number of threads (how many should always be running) than the "max" (maximum number that the pool can grow too) and how long the threads should start dormant before they should exit to keep the thread count down at the "core" number. The problem with this is that then you'll need to have a fixed size job queue otherwise the 2nd thread will never be created. That's (unfortunately) how the ThreadPoolExecutor works.

If you have a different core and max thread numbers and you are submitting a large number of jobs to your thread-pool then you'll need to block the producer else the jobs will be rejected by the queue if it fills up.

Something like:

ThreadPoolExecutor threadPool = new ThreadPoolExecutor(CORE_POOL_SIZE, MAX_POOL_SIZE,
    60, TimeUnit.SECONDS, new LinkedBlockingQueue<>(MAX_QUEUE_SIZE));
// need to say what to do if the queue is full
threadPool.setRejectedExecutionHandler(new RejectedExecutionHandler() {
     public void rejectedExecution(Runnable r, ThreadPoolExecutor executor) {
          // this will block the caller if the queue is full
          executor.getQueue().put(r);
     }
});

Answer 4

better to shutdown the executors once it's done. it will release all the threads which are created with executor Service.

finally {
        if(!executors.isShutdown())
            executors.shutdown();
    }