Catching Exceptions from (ExecutorService) CachedThreadPool

Question

We are using a CachedThreadPool created via the ExecutorService#newCachedThreadPool. (Java 1.6). We are getting errors elsewhere in our code: "unable to create new native thread" which we've diagnosed as an out of memory problem. When this problem is happening, the block of code where we call submit (we use submit, not execute) slows down. We suspect that the ThreadPool is having the same issue "unable to create new native thread" when it is trying to create new threads to process the tasks, but we can't be sure.

Is there any way to catch exceptions from inside the ExecutorService? To be 100% clear, I'm not talking about the tasks given to the ExecutorService, but from the ExecutorService itself.

Maybe you should do a thread dump and see where all those threads are going. This shouldn't be happening. — Boris the Spider, Apr 10 '13 at 23:25
This is definitely a system load issue. I get it shouldn't be going. It's due to a downstream dependency failure and us having retry logic on our side. Threads get all queued up waiting to retry. — John Hinnegan, Apr 11 '13 at 21:14

score 3 · Accepted Answer · answered Apr 11 '13 at 01:51

The Executors.newCachedThreadPool creates a ThreadPoolExecutor with a unlimited maximum thread pool size(note the second argument of ThreadPoolExecutor constructor):

public static ExecutorService newCachedThreadPool() {
    return new ThreadPoolExecutor(0, Integer.MAX_VALUE,
                                  60L, TimeUnit.SECONDS,
                                  new SynchronousQueue<Runnable>());
}

That means if you are submitting tasks faster than the consuming rate, new threads will be created for each new tasks, eventually it will reach the system limit and the "unable to create new native thread" exception is thrown.

To fix this, you need to change the configuration of ThreadPoolExecutor:

Use a reasonable ThreadPoolExecutor.
Choose a proper reject policy when the ThreadPoolExecutor is exhausted.

For example:

ExecutorService executorService = new ThreadPoolExecutor(5,200,
                          60L, TimeUnit.SECONDS,
                          new ArrayBlockingQueue(1000),
                          Executors.defaultThreadFactory(),
                          new ThreadPoolExecutor.CallerRunsPolicy());

Please read the JavaDocs for configuration details.

Thanks. I have been digging through the docs, and I get why it happens, but I was wondering if there was any way to catch the error? Would I have to write my own ThreadFactory? — John Hinnegan, Apr 11 '13 at 21:15
@JohnHinnegan I don't understand why you want to catch the error. I think it is a wrong design to use an thread pool of unlimited pool size for long running tasks. — ericson, Apr 12 '13 at 02:21
they're not long running. I'm not disagreeing that an unbounded pool is risky or just plain wrong. But a fixed pool could have this problem if something else on the host was F*ing up. I'm trying to build robust monitoring into my systems, and this error cropped up and went undetected in 1 part of our system. I know this was the error because we caught it elsewhere (and obviously it was global to the JVM). When you answered, I didn't grok what a RejectedExecutionHandler was. I think this answers my question. I think I could implement one of these to handle the exception. — John Hinnegan, Apr 12 '13 at 13:43

score 0 · Answer 2 · answered Aug 23 '13 at 21:51

You could use a custom class that implements Runnable with a collection of excpetions inside, like this:

public class MyRunnable implements Runnable {
    private List<Exception> exceptions;

    ...

    public void addException(Exception e) { ... }
    public void getExceptions(){ ... }
}

After all runnables finish their executions you could check for exceptions inside them and throw another exception in response.

Catching Exceptions from (ExecutorService) CachedThreadPool

2 Answers2