Executors.newCachedThreadPool() versus Executors.newFixedThreadPool()

Question

newCachedThreadPool() versus newFixedThreadPool()

When should I use one or the other? Which strategy is better in terms of resource utilization?

Answer 1

I think the docs explain the difference and usage of these two functions pretty well:

newFixedThreadPool

Creates a thread pool that reuses a fixed number of threads operating off a shared unbounded queue. At any point, at most nThreads threads will be active processing tasks. If additional tasks are submitted when all threads are active, they will wait in the queue until a thread is available. If any thread terminates due to a failure during execution prior to shutdown, a new one will take its place if needed to execute subsequent tasks. The threads in the pool will exist until it is explicitly shutdown.

newCachedThreadPool

Creates a thread pool that creates new threads as needed, but will reuse previously constructed threads when they are available. These pools will typically improve the performance of programs that execute many short-lived asynchronous tasks. Calls to execute will reuse previously constructed threads if available. If no existing thread is available, a new thread will be created and added to the pool. Threads that have not been used for sixty seconds are terminated and removed from the cache. Thus, a pool that remains idle for long enough will not consume any resources. Note that pools with similar properties but different details (for example, timeout parameters) may be created using ThreadPoolExecutor constructors.

In terms of resources, the newFixedThreadPool will keep all the threads running until they are explicitly terminated. In the newCachedThreadPool Threads that have not been used for sixty seconds are terminated and removed from the cache.

Given this, the resource consumption will depend very much in the situation. For instance, If you have a huge number of long running tasks I would suggest the FixedThreadPool. As for the CachedThreadPool, the docs say that "These pools will typically improve the performance of programs that execute many short-lived asynchronous tasks".

Answer 2

Just to complete the other answers, I would like to quote Effective Java, 2nd Edition, by Joshua Bloch, chapter 10, Item 68 :

"Choosing the executor service for a particular application can be tricky. If you’re writing a small program, or a lightly loaded server, using Executors.new- CachedThreadPool is generally a good choice, as it demands no configuration and generally “does the right thing.” But a cached thread pool is not a good choice for a heavily loaded production server!

In a cached thread pool, submitted tasks are not queued but immediately handed off to a thread for execution. If no threads are available, a new one is created. If a server is so heavily loaded that all of its CPUs are fully utilized, and more tasks arrive, more threads will be created, which will only make matters worse.

Therefore, in a heavily loaded production server, you are much better off using Executors.newFixedThreadPool, which gives you a pool with a fixed number of threads, or using the ThreadPoolExecutor class directly, for maximum control."

Answer 3

If you see the code in the grepcode, you will see, they are calling ThreadPoolExecutor. internally and setting their properties. You can create your one to have a better control of your requirement.

public static ExecutorService newFixedThreadPool(int nThreads) {
   return new ThreadPoolExecutor(nThreads, nThreads,0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<Runnable>());
}

public static ExecutorService newCachedThreadPool() {
        return new ThreadPoolExecutor(0, Integer.MAX_VALUE,
                                      60L, TimeUnit.SECONDS,
                                      new SynchronousQueue<Runnable>());
}

Answer 4

The ThreadPoolExecutor class is the base implementation for the executors that are returned from many of the Executors factory methods. So let's approach Fixed and Cached thread pools from ThreadPoolExecutor's perspective.

ThreadPoolExecutor

The main constructor of this class looks like this:

public ThreadPoolExecutor(
                  int corePoolSize,
                  int maximumPoolSize,
                  long keepAliveTime,
                  TimeUnit unit,
                  BlockingQueue<Runnable> workQueue,
                  ThreadFactory threadFactory,
                  RejectedExecutionHandler handler
)

Core Pool Size

The corePoolSize determines the minimum size of the target thread pool. The implementation would maintain a pool of that size even if there are no tasks to execute.

Maximum Pool Size

The maximumPoolSize is the maximum number of threads that can be active at once.

After the thread pool grows and becomes bigger than the corePoolSize threshold, the executor can terminate idle threads and reach to the corePoolSize again. If allowCoreThreadTimeOut is true, then the executor can even terminate core pool threads if they were idle more than keepAliveTime threshold.

So the bottom line is if threads remain idle more than keepAliveTime threshold, they may get terminated since there is no demand for them.

Queuing

What happens when a new task comes in and all core threads are occupied? The new tasks will be queued inside that BlockingQueue<Runnable> instance. When a thread becomes free, one of those queued tasks can be processed.

There are different implementations of the BlockingQueue interface in Java, so we can implement different queuing approaches like:

1. Bounded Queue: New tasks would be queued inside a bounded task queue.

2. Unbounded Queue: New tasks would be queued inside an unbounded task queue. So this queue can grow as much as the heap size allows.

3. Synchronous Handoff: We can also use the SynchronousQueue to queue the new tasks. In that case, when queuing a new task, another thread must already be waiting for that task.

Work Submission

Here's how the ThreadPoolExecutor executes a new task:

1. If fewer than corePoolSize threads are running, tries to start a new thread with the given task as its first job.
2. Otherwise, it tries to enqueue the new task using the BlockingQueue#offer method. The offer method won't block if the queue is full and immediately returns false.
3. If it fails to queue the new task (i.e. offer returns false), then it tries to add a new thread to the thread pool with this task as its first job.
4. If it fails to add the new thread, then the executor is either shut down or saturated. Either way, the new task would be rejected using the provided RejectedExecutionHandler.

The main difference between the fixed and cached thread pools boils down to these three factors:

1. Core Pool Size
2. Maximum Pool Size
3. Queuing

+-----------+-----------+-------------------+---------------------------------+
| Pool Type | Core Size |    Maximum Size   |         Queuing Strategy        |
+-----------+-----------+-------------------+---------------------------------+
|   Fixed   | n (fixed) |     n (fixed)     | Unbounded `LinkedBlockingQueue` |
+-----------+-----------+-------------------+---------------------------------+
|   Cached  |     0     | Integer.MAX_VALUE |        `SynchronousQueue`       |
+-----------+-----------+-------------------+---------------------------------+

---

Fixed Thread Pool

---

Here's how the Excutors.newFixedThreadPool(n) works:

public static ExecutorService newFixedThreadPool(int nThreads) {
    return new ThreadPoolExecutor(nThreads, nThreads,
                                  0L, TimeUnit.MILLISECONDS,
                                  new LinkedBlockingQueue<Runnable>());
}

As you can see:

• The thread pool size is fixed.
• If there is high demand, it won't grow.
• If threads are idle for quite some time, it won't shrink.
• Suppose all those threads are occupied with some long-running tasks and the arrival rate is still pretty high. Since the executor is using an unbounded queue, it may consume a huge part of the heap. Being unfortunate enough, we may experience an OutOfMemoryError.

When should I use one or the other? Which strategy is better in terms of resource utilization?

A fixed-size thread pool seems to be a good candidate when we're going to limit the number of concurrent tasks for resource management purposes.

For example, if we're going to use an executor to handle web server requests, a fixed executor can handle the request bursts more reasonably.

For even better resource management, it's highly recommended to create a custom ThreadPoolExecutor with a bounded BlockingQueue<T> implementation coupled with reasonable RejectedExecutionHandler.

---

Cached Thread Pool

---

Here's how the Executors.newCachedThreadPool() works:

public static ExecutorService newCachedThreadPool() {
    return new ThreadPoolExecutor(0, Integer.MAX_VALUE,
                                  60L, TimeUnit.SECONDS,
                                  new SynchronousQueue<Runnable>());
}

As you can see:

• The thread pool can grow from zero threads to Integer.MAX_VALUE. Practically, the thread pool is unbounded.
• If any thread is idle for more than 1 minute, it may get terminated. So the pool can shrink if threads remain too much idle.
• If all allocated threads are occupied while a new task comes in, then it creates a new thread, as offering a new task to a SynchronousQueue always fails when there is no one on the other end to accept it!

When should I use one or the other? Which strategy is better in terms of resource utilization?

Use it when you have a lot of predictable short-running tasks.

Answer 5

If you are not worried about an unbounded queue of Callable/Runnable tasks, you can use one of them. As suggested by bruno, I too prefer newFixedThreadPool to newCachedThreadPool over these two.

But unbounded queue size is always dangerous. If you have some unexpected turbulence in the system and current threads are stuck, the queue size is going to increase and it may go out of bounds by causing OutOfMemory error OR degraded performance in the system.

But ThreadPoolExecutor provides more flexible features compared to either newFixedThreadPool or newCachedThreadPool. It provides more granular control on various attributes.

ThreadPoolExecutor(int corePoolSize, int maximumPoolSize, long keepAliveTime, 
TimeUnit unit, BlockingQueue<Runnable> workQueue, ThreadFactory threadFactory,
RejectedExecutionHandler handler)

Advantages:

1. You have full control of BlockingQueue size. It's not un-bounded, unlike the earlier two options. I won't get an out of memory error due to a huge pile-up of pending Callable/Runnable tasks when there is unexpected turbulence in the system.

2. You can implement custom Rejection handling policy OR use one of the policies:

   1. In the default ThreadPoolExecutor.AbortPolicy, the handler throws a runtime RejectedExecutionException upon rejection.

   2. In ThreadPoolExecutor.CallerRunsPolicy, the thread that invokes execute itself runs the task. This provides a simple feedback control mechanism that will slow down the rate that new tasks are submitted.

   3. In ThreadPoolExecutor.DiscardPolicy, a task that cannot be executed is simply dropped.

   4. In ThreadPoolExecutor.DiscardOldestPolicy, if the executor is not shut down, the task at the head of the work queue is dropped, and then execution is retried (which can fail again, causing this to be repeated.)

3. You can implement a custom Thread factory for the below use cases:

   By supplying a different ThreadFactory, you can alter the thread's name, thread group, priority, daemon status, etc

Answer 6

That’s right, Executors.newCachedThreadPool() isn't a great choice for server code that's servicing multiple clients and concurrent requests.

Why? There are basically two (related) problems with it:

1. It's unbounded, which means that you're opening the door for anyone to cripple your JVM by simply injecting more work into the service (DoS attack). Threads consume a non-negligible amount of memory and also increase memory consumption based on their work-in-progress, so it's quite easy to topple a server this way (unless you have other circuit-breakers in place).

2. The unbounded problem is exacerbated by the fact that the Executor is fronted by a SynchronousQueue which means there's a direct handoff between the task-giver and the thread pool. Each new task will create a new thread if all existing threads are busy. This is generally a bad strategy for server code. When the CPU gets saturated, existing tasks take longer to finish. Yet more tasks are being submitted and more threads created, so tasks take longer and longer to complete. When the CPU is saturated, more threads is definitely not what the server needs.

Here are my recommendations:

Use a fixed-size thread pool Executors.newFixedThreadPool or a ThreadPoolExecutor. with a set maximum number of threads;

Answer 7

You must use newCachedThreadPool only when you have short-lived asynchronous tasks as stated in Javadoc, if you submit tasks which takes longer time to process, you will end up creating too many threads. You may hit 100% CPU if you submit long running tasks at faster rate to newCachedThreadPool (http://rashcoder.com/be-careful-while-using-executors-newcachedthreadpool/).

Answer 8

I do some quick tests and have the following findings:

1) if using SynchronousQueue:

After the threads reach the maximum size, any new work will be rejected with the exception like below.

Exception in thread "main" java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3fee733d rejected from java.util.concurrent.ThreadPoolExecutor@5acf9800[Running, pool size = 3, active threads = 3, queued tasks = 0, completed tasks = 0]

at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)

2) if using LinkedBlockingQueue:

The threads never increase from minimum size to maximum size, meaning the thread pool is fixed size as the minimum size.