Handling the Hanging Tasks

Question

This is just an example to explain my problem...

I am using ExecutorService with 20 active threads and 75K max queued items...

In my case, a normal task should not take more than 10 seconds, if it takes more time that means there's some problem with the task.

If all the threads are hung due to problematic tasks my RejectionHandler would restart the entire service.

I have two questions here:

1. I do not like the idea of restarting the service, instead if there's way to detect hanging thread and we could just restart that hung thread that would be great. I have gone through couple of articles to handle hung threads with ThreadManager but have not found anything with ExecutorService.

2. I am very much fascinated about the Executors.newCachedThredPool() because on peak days we are heavily loaded with incoming tasks, and on other days they are very few. Any suggestions would be greatly appreciated.

   public class HangingThreadTest {

    // ExecutorService executorService = Executors.newCachedThreadPool()
    private static ExecutorService  executorService = new ThreadPoolExecutor(10,
            20, 5L, TimeUnit.SECONDS, new LinkedBlockingQueue<Runnable>(75000));

    public static void main(String... arg0) {
        for (int i = 0; i < 50000; i++) {

            executorService.submit(new Task());
        }
    }

}

/**
 * Task to be completed
 */
class Task implements Runnable {

    private static int count = 0;

    @Override
    public void run() {

        count++;
        if (count%5 == 0) {
            try {
                System.out.println("Hanging Thread task that needs to be reprocessed: "
                        + Thread.currentThread().getName()+" count: "+count);
                Thread.sleep(11000);
            } catch (InterruptedException e) {
                // Do something
            }
        }
        else{
            System.out.println("Normal Thread: "
                    + Thread.currentThread().getName()+" count: "+count);
            try {
                Thread.sleep(1000);
            } catch (InterruptedException e) {
                //Do something
            }
        }

    }

}

Answer 1

There is no build-in mechanism in Executors framework that would help terminate a thread if it has been running for more than a threshold value.

But we can achieve this with some extra code as below:

1. Get the Future object returned by the executorService.submit(...);.

Future future = executorService.submit(new Task());

2. Call the get method on this future object to and make it wait only for threshold interval for task completion. Below, an example that is waits for only 2 secs.

   try {
           f.get(2, TimeUnit.SECONDS);
       } catch (TimeoutException e) {
           f.cancel(true);
       } catch (Exception e) {}
   

The above code waits for 2 seconds for task completion it throws a TimeoutException if it doesn't get completed during that time. Subsequently we can call cancel method on the future object. This results in setting the interrupt flag in the thread that is executing the task.

3. Now the final change is, in the Task class code we need to check at necessary points (application dependent), whether the interrupt flag has been set to true using isInterrupted() method of Thread class. If interrupted==true, we can do the necessary clean up and return from the run method immediately. The critical piece here is to identify the necessary points in your Task class where you want to check for this interrupted flag.

This makes the thread available for processing next task.

Answer 2

You may have a look at this article, it was very helpful for me before when I was facing the same problem : Java Hanging Thread Detection