Correct way to use a Google Apps Marketplace service account to connect to Gmail IMAP and other services

Question

One of the features of our Marketplace app makes use of accessing the user's Gmail account via IMAP. We are using the google-api-java-client and google-oauth-java-client libraries and code similar to this example in the java-gmail-imap project as follows:

GoogleCredential credential = new GoogleCredential.Builder().setTransport(HTTP_TRANSPORT)
    .setJsonFactory(JSON_FACTORY)
    .setServiceAccountId(SERVICE_ACCOUNT_ID)
    .setServiceAccountScopes(Arrays.asList(GMAIL_SCOPE))
    .setServiceAccountPrivateKey(PRIVATE_KEY)
    .setServiceAccountUser(emailAddress)
    .build();
credential.refreshToken();

We are then using code based on the examples at https://code.google.com/p/google-mail-oauth2-tools to make the IMAP connection e.g.

IMAPStore imapStore = OAuth2Authenticator.connectToImap("imap.googlemail.com",
    993, emailAddress, credential.getAccessToken(), false);

The majority of the time this appears to work correctly, however we are seeing that for a small but significant number of requests the call to Google made by refreshToken() fails with an HTTP 500 error and an HTML response where the JSON would normally be returned e.g.

<p class="large"><b>500.</b> <ins>That's an error.</ins></p>
<p class="large">The server could not process your request.
<ins>That's all we know.</ins></p>

We were advised by a developer advocate at Google that we refresh tokens are not supported for service accounts and we should be using an approach like in this example.

However, it seems like without the call to refreshToken then accessToken is not populated on the credential object and then this results in a NullPointerException when we call OAuth2Authenticator.connectToImap

From the source for GoogleCredential it did seem like executeRefreshToken() is overridden to handle service accounts i.e. instead of performing a refresh it simply requests a new token, and then this bit of code in Credential then handles populating the access token:

TokenResponse tokenResponse = executeRefreshToken();
if (tokenResponse != null) {
    setFromTokenResponse(tokenResponse); ....

We were unsure whether we need to enclose our call to refreshToken() in a retry loop to work around the intermittent 500 errors or whether we need to make other changes to our code to follow the recommended approach for this scenario.

Can anyone advise?

Answer 1

I use the java-gmail-imap example code in production (but it is only used to display an inbox in our University portal, there isn't much interaction that would require me to reuse the same refresh token for instance).

Depending on your usage, I wonder if in your case some kind of throttling is coming into play (I've read in places that Gmail can occasionally throttle access).

Elsewhere I've seen Google APIs talk about making retries using an exponential backoff algorithm.

You have to be a little careful when comparing the usage of OAuth 2.0 with the other Google Service APIs and Gmail. Gmail is special in that it uses XOAUTH2. That said I've seen other Google API's that appear to need the refreshToken call. The documentation is a bit unclear and says things like "Refresh the access token, if necessary" (as you say it doesn't seem to work without this step but I haven't done any experimentation with re-using refresh tokens via credential.setRefreshToken(String refreshToken)).

I'd be interested to hear how you get on.