How to debug CompletableStage deadlocks?

Question

The most difficult debugging problem I've run across recently is deadlocks between asynchronous operations. For example, given two CompletionStage chains, where the first chain invokes a method that depends upon the completion of the second chain, and the second chain invokes a method that depends upon the completion of the first chain. It isn't this obvious in real-life because the dependency tends to be hidden and sometimes deadlocks involve more than three parties.

Part of the problem is that there is no way to find out what a CompletableStage is waiting on. This is because an operation references a CompletableStage, not the other way around.

Most debuggers provide some level of deadlock detection nowadays, but this only applies to threads. How does one debug deadlocks between CompletableStage chains?

Could you give an example where this situation happens? I think there are 2 possible situations but I might miss some: when you manually manage a `CompletableFuture` with `new`/`complete()`, and when one of the tasks in the chain calls `get()`/`join()` on a descendant stage. The first case seems quite difficult to detect (probably requires code analysis) and the second one seems a bit contrived. Is it something often happening in real life? It seems it would be more likely to have deadlocks involving other mechanisms (`synchronized`, database transactions etc.). — Didier L, May 02 '18 at 15:58

score 1 · Accepted Answer · answered May 15 '18 at 14:09

I ended up doing the following:

At the end of each CompletionStage chain, schedule an event that will get fired after a timeout:

Set<Object> knownDeadlocks = ConcurrentHashMap.newKeySet();
// ...
Future<?> deadlockListener = scope.getScheduler().schedule(() ->
{
    if (knownDeadlocks.add(Throwables.getStackTraceAsString(context)))
        log.warn("Possible deadlock", context);
}, DEADLOCK_DURATION.toMillis(), TimeUnit.MILLISECONDS);

Use CompletionStage.handle() to disable deadlockListener if the stage completes as expected:

return stage.handle((value, throwable) ->
{
    // WARNING: By design, CompletionStage.whenComplete() suppresses any exceptions thrown by its argument, so we use handle() instead.
    deadlockListener.cancel(false);
    if (throwable == null)
        return value;
    return rethrowException(throwable);
});

For completeness, you also have:

/**
 * Rethrows a {@code Throwable}, wrapping it in {@code CompletionException} if it isn't already wrapped.
 *
 * @param <T>       the return type expected by the caller
 * @param throwable a Throwable
 * @return an undefined value (the method always throws an exception)
 * @throws CompletionException wraps {@code throwable}
 */
public <T> T rethrowException(Throwable throwable)
{
    if (throwable instanceof CompletionException)
        throw (CompletionException) throwable;
    if (throwable == null)
        throwable = new NullPointerException("throwable may not be null");
    // According to https://stackoverflow.com/a/49261367/14731 some methods do not wrap exceptions
    throw new CompletionException(throwable);
}

How to debug CompletableStage deadlocks?

1 Answers1

Linked