Garbage collection seems to shutdown local executor and cause RejectedExecutionException

Question

Need help on an intermittent headache. The code is calling com.google.api.client.http.HttpRequest#executeAsync() which basically has the following logic,

  @Beta
  public Future<HttpResponse> executeAsync(Executor executor) {
    FutureTask<HttpResponse> future = new FutureTask<HttpResponse>(new Callable<HttpResponse>() {

      public HttpResponse call() throws Exception {
        return execute();
      }
    });
    executor.execute(future);
    return future;
  }

  @Beta
  public Future<HttpResponse> executeAsync() {
    return executeAsync(Executors.newSingleThreadExecutor());
  }

The call runs into java.util.concurrent.RejectedExecutionException sometimes and from the log seems when this happens it's always accompanied by garbage collection. Below is a sample of the log pattern when this happens,

2017-09-26 11:04:56.039186 2017-09-26T11:04:56.012+0000: [GC pause (G1 Evacuation Pause) (young) 213M->50M(300M), 0.0262262 secs]
2017-09-26 11:04:56.048210 java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@71a0a39 rejected from java.util.concurrent.ThreadPoolExecutor@36c306aa[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2017-09-26 11:04:56.048212      at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) ~[?:1.8.0_141]
2017-09-26 11:04:56.048214      at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_141]
2017-09-26 11:04:56.048216      at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_141]
2017-09-26 11:04:56.048218      at java.util.concurrent.Executors$DelegatedExecutorService.execute(Executors.java:668) ~[?:1.8.0_141]
2017-09-26 11:04:56.048220      at com.google.api.client.http.HttpRequest.executeAsync(HttpRequest.java:1085) ~[google-http-client-1.21.0.jar:1.21.0]
2017-09-26 11:04:56.048222      at com.google.api.client.http.HttpRequest.executeAsync(HttpRequest.java:1099) ~[google-http-client-1.21.0.jar:1.21.0]

Based on my understanding GC shouldn't clean up this executor since it hasn't gone out of scope yet, but judging by the log of the error this seems to be the behavior.

EDIT: Thanks for the quick responses. I probably need to clarify I cannot reproduce this either at my will otherwise I probably will have more clue. The code generally runs fine but this error does happen rarely and all we have is the log information I pasted for the symptom.

EDIT2: Clarification the code I pasted is not mine. It's the open source library google-http-java-client I'm using. So I have no way to change that. I sure can instead create a long-term Executor myself and call it with the first method, but I'm trying to understand what's the problem for now.

What shuts down your `Executor`? If you're spamming thousands of sleeping threads then having issues is very likely. — Boris the Spider, Sep 27 '17 at 21:45
Are you somehow submitting a new task after having called shutdown() on the Executor? — jwils, Sep 27 '17 at 21:53
@SotiriosDelimanolis, I probably meant life cycle, or sorry maybe I'm just bad at official terms. I thought in this case once `executeAsync()` is finished the executor is free for GC to take (shutdown is called inside this executor's finalizer). But when the exception happens the executor is still in use so GC should by no means touch it and cause it to shutdown right? — Derek, Sep 27 '17 at 22:11
@jwils please note this part is not controlled by me, rather the google http client I'm using. As a user I simply call that `executeAsync()` in my code and the executor used is completely inside this `HttpRequest` class. — Derek, Sep 27 '17 at 22:13

Sotirios Delimanolis · Accepted Answer · 2019-02-15T17:41:42.450

The ExecutorService returned by Executors#newSingleThreadExecutor() happens to be ThreadPoolExecutor wrapped in a FinalizableDelegatedExecutorService, an implementation detail that shuts down its wrapped ExecutorService on finalization. We can tell the executor is shutdown from your logs

[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]

Why is the object being finalized, though? You state

Based on my understanding GC shouldn't clean up this executor since it hasn't gone out of scope yet, but judging by the log of the error this seems to be the behavior.

This is a common misconception. Scope is a compile time feature that determines where a name can be used to refer to some entity in source code. It has no bearing on runtime garbage collection.

Garbage collection (and finalization) is controlled by reachability of objects. The Java Language Specification states

When an object is no longer referenced, it may be reclaimed by the garbage collector. If an object declares a finalizer, the finalizer is executed before the object is reclaimed to give the object a last chance to clean up resources that would not otherwise be released. When a class is no longer needed, it may be unloaded.

The JVM will not garbage collect reachable objects

A reachable object is any object that can be accessed in any potential continuing computation from any live thread.

As explained in the accepted answer (by an Oracle developer) for this question

finalize() called on strongly reachable object in Java 8

reachability analysis allows

for an object to be finalized and garbage collected even if there are references to it in local variables on the stack

What you're seeing is the JVM making the decision that the FinalizableDelegatedExecutorService (and its ThreadPoolExecutor) is no longer reachable by a continuing computation from a live thread and finalizing it. That action shuts down the executor and the RejectedExecutionException is thrown when a task is submitted.

This is a known issue for that implementation and bug JDK-8145304 was opened to discuss solutions (essentially just better documentation).

If it was up to me, google-http-client would change their implementation to use Executors.newFixedThreadPool(1) which isn't wrapped with a FinalizableDelegatedExecutorService. (I've opened this issue to discuss a solution for the library.)

My suggestion is that you create your own ExecutorService (maybe with newFixedThreadPool) and use the overloaded executeAsync(Executor) method.

The issue has been fixed, see here.

Came for an answer, got an answer, plus a mini lesson. Wish I could give you two accepts, thanks! — Derek, Sep 27 '17 at 22:32
There’s also [this statement in the specification](https://docs.oracle.com/javase/specs/jls/se8/html/jls-12.html#jls-12.6.1): “*Optimizing transformations of a program can be designed that reduce the number of objects that are reachable to be less than those which would naively be considered reachable. […] Another example of this occurs if the values in an object's fields are stored in registers. The program may then access the registers instead of the object, and never access the object again. This would imply that the object is garbage. […]*” — Holger, Sep 28 '17 at 10:58
By the way, if I know beforehand that only one job will get scheduled, I’d rather use `executeAsync(r -> new Thread(r).start());` which will imply automatic cleanup, without the risk of early finalization… — Holger, Sep 28 '17 at 11:02
[reachabilityFence](http://download.java.net/java/jdk9/docs/api/java/lang/ref/Reference.html#reachabilityFence-java.lang.Object-) is another option to counteract those optimizations — the8472, Sep 28 '17 at 12:53

Garbage collection seems to shutdown local executor and cause RejectedExecutionException

1 Answers1