How do I kill a Java Future?

Question

The service I'm working on uses a Future to run multiple tasks in parallel; each task can take up to a minute to complete. However, it seems the external lib is buggy, since in some occasions (2% of the time) it doesn't return. In those cases I would like to give a 2-minute wait time, and if it hasn't returned, I would like to kill the future and re-schedule again later (it will succeed eventually).

How do I kill the Future?

  private void run() {
    ExecutorService queue = Executors.newFixedThreadPool(1);

    Future<Integer> f = queue.submit(new MyTask());
    Thread.sleep(500);

    try {
      Integer r = f.get(120, TimeUnit.SECONDS);
    } catch (InterruptedException | ExecutionException | TimeoutException e) {
      e.printStackTrace();
      f.cancel(true);
    }

    // Bad future still running here and I need it dead.

  }

  private class MyTask implements Callable<Integer> {
    private ExternalLibrary extlib = new ExternalLibrary();

    @Override
    public Integer call() throws Exception {
      // step 1 - do a few things

      // step 2 - process data
      Integer val = this.extlib.doSomething(); // here's the problem!

      // step 3 - do other things

      return val;
    }

  }

I can see the external lib running and consuming CPU (for 24 hours)... doing nothing. It's a simple task that should never take more than 60 seconds to complete its work.

So far, I'm killing the whole JVM once a day to get rid of this issue, but I'm sure there must be a better way. I wonder how app servers (Tomcat, JBoss, Weblogic, etc.) do it with rogue processes.

Answer 1

Even if you could kill the future hanging in the buggy library, this does likely not solve your problem. The library might still have acquired some resource which will not be properly clean up. This might be memory allocations, open file handles or even monitors leaving some internal data structures in an inconsistent state. Eventually you will likely be back at the point where you have to restart your JVM.

There's basically two options: Fix or isolate it.

1. Fix: try to get the library fixed. If this is not possible,
2. isolate: isolate the library into a external service your application depends on. E.g. implement a REST API for calling the library and wrap everything up into a Docker image. Automate restarting of the Docker container as needed.

Answer 2

As others have mentioned, stopping a Future is cooperative, meaning, the thread running async must respond to cancellation from the waiting thread. If the async task isn't cooperative simply invoking shutdown or shutdownNow won't be enough as the underlying TPE will just interrupt the threads.

If you have no control over extlib, and extlib is not cooperative, I see two options

1. You can stop the thread currently running. This can cause issues if the thread being stopped currently is holding a lock or some other resource. It can lead to interesting bugs that could be hard to dissect.
2. This could take some more work, but you could run the async task as a separate process entirely. The TPE can still run the process and, on interruption, can destroy the process. This obviously has more interesting issues like how to load the process with required input.

Answer 3

If I understand your requirement correctly & based on your requirement (i.e. 1 thread), you can look for shutting down executorservice in 2 phases, code is available in java doc of executorservice:

try {
      Integer r = f.get(120, TimeUnit.SECONDS);
    } catch (InterruptedException | ExecutionException | TimeoutException e) {
      e.printStackTrace();
      //f.cancel(true);  you can omit this call if you wish. 
      shutdownAndAwaitTermination(queue);
    } ... //remaining method code


void shutdownAndAwaitTermination(ExecutorService pool) {
   pool.shutdown(); // Disable new tasks from being submitted
   try {
     // Wait a while for existing tasks to terminate
     if (!pool.awaitTermination(60, TimeUnit.SECONDS)) {
       pool.shutdownNow(); // Cancel currently executing tasks
       // Wait a while for tasks to respond to being cancelled
       if (!pool.awaitTermination(60, TimeUnit.SECONDS))
           System.err.println("Pool did not terminate");
     }
   } catch (InterruptedException ie) {
     // (Re-)Cancel if current thread also interrupted
     pool.shutdownNow();
     // Preserve interrupt status
     Thread.currentThread().interrupt();
   }
 }

Please read documentation about shutdown() , shutdownNow() how they behaves because it clearly mentions there is no 100% guarantee that tasks / executorservice will get stopped if its running.

Answer 4

Unfortunately if the external library is not co-operating to thread interrupts, there is nothing you can do to kill the Thread running the task managed by the ExecutorService.

An alternative that I can think of is to run the offending code as a separate process. Using ProcessBuilder and Process, your task can effectively control (or) even kill the offending process after a timeout (https://docs.oracle.com/javase/9/docs/api/java/lang/Process.html#destroyForcibly--).

Also see https://docs.oracle.com/javase/9/docs/api/java/lang/ProcessBuilder.html

Answer 5

@joe That is correct. Unless you have control over the thread and inside the thread you can't kill it.

this.extlib.doSomething();

if this line starts a thread then we need to get hold of that thread to kill it as we don't have reference to stop it.

Answer 6

In your code, the call:

this.extlib.doSomething()

must be synchronous, because if it is not, the code lost sense. With that assumption, you can try:

ExecutorService executor = Executors.newSingleThreadExecutor();
Future<Integer> future = executor.submit(new MyTask());
try {
    future.get(120, TimeUnit.SECONDS);
} catch (InterruptedException | ExecutionException e) {
    e.printStackTrace();
} catch (TimeoutException e) {
    future.cancel(true);
} finally {
    executor.shutdownNow();
}

If this doesn't stop the doSomethig work is because this doSomething function is opening other threads to do the work. In that case, maybe you can check the threads that are running with:

Thread.getAllStackTraces()

And try to kill the right one...