GCM (legacy FCM) sends GOAWAY HTTP/2 frames

Question

Our push backend uses GCM API to send notifications to Android devices.
Endpoint: https://fcm.googleapis.com/fcm/send
HTTP client: Java 11 Http Client (with HTTP/2 setting):

var httpClient = HttpClient.newBuilder()
    .version(HttpClient.Version.HTTP_2)
    // ...
    .build();

Problem
Sometimes we get GOAWAY frames from GCM. But java http client hides both error-code and payload for those frames (just throws some generic exception). So we don't know what is causing GOAWAY.

Questions

1. Is it an expected behavior of GCM API?
2. What could cause it?
3. How to debug it?

Answer 1

It's hard to tell for sure why GCM sends GOAWAY - only the GCM team can really answer this. However some of the usual reasons for it are:

• The server your client is connected to is going down for maintainence. Before doing that it enters a draining mode that allows to complete ongoing requests while preventing new ones. This is achieved by sending a GOAWAY that informs the client that no new requests should be started on that connection.
• The server wants the client to migrate to a different connection for load balancing purposes

That means a server sending a GOAWAY is not a bug, it's just something to be expected in a cloud setup.

The HTTP/2 specifciation provides some further information on how to handle GOAWAY: https://www.rfc-editor.org/rfc/rfc7540#section-6.8

Regarding handling it: Ideally your HTTP/2 client might already do that fully internally:

• if it receives a GOAWAY frame while no request is active it should create a new connection for the next request
• if it receives a GOAWAY frame while a request has been sent, and if that GOAWAY frame indicates the request hasn't been processed, a new client can be created and the request retried. This works even if its a non-idempotent request (like a POST), since we know the server hasn't processed the request yet.

If the JDK11 client does not do that automatically, the only thing you can really do here is retry on application layer.

According to is there any way to handle HTTP/2 Goaway received IOException in HttpClient java? it seems like the client might throw an IOException whose message contains GOAWAY received. That might be a good signal to retry.