Catching GoogleJsonResponseException but e.getDetails() returns null

Question

I have a non-Android Java app uploading thousands of files, one after the other.

For each insert, I'm implementing exponential back off like so:

Random randomGenerator = new Random();
        for(int i = 0; i < 5; i++) {
            try {
                return service.files().insert(body, mediaContent).execute();
            } catch(GoogleJsonResponseException e) {
                if (e.getDetails().getCode() == 500
                        || e.getDetails().getCode() == 501
                        || e.getDetails().getCode() == 502
                        || e.getDetails().getCode() == 503
                        || e.getDetails().getCode() == 504
                        || e.getDetails().getCode() == 403) {
                    Thread.sleep((1 << i) * 1000
                            + randomGenerator.nextInt(1001));
                } else {
                    // error besides 5xx
                    throw e;
                } 
            } catch (HttpResponseException e) {
                if (e.getStatusCode() == 500 || e.getStatusCode() == 501
                        || e.getStatusCode() == 502 || e.getStatusCode() == 503
                        || e.getStatusCode() == 504
                        || e.getStatusCode() == 403) {
                    Thread.sleep((1 << i) * 1000
                            + randomGenerator.nextInt(1001));
                } else {
                    // error besides 5xx
                    throw e;
                }
            } catch (SocketTimeoutException e) {
                Thread.sleep((1 << i) * 1000 + randomGenerator.nextInt(1001));
            }

Here's the issue:

There have been multiple times in which my app catches a GoogleJsonResponseException, yet e.getDetails() returns null thus I get null pointer exception when checking the error code.

per the google docs at http://javadoc.google-api-java-client.googlecode.com/hg/1.6.0-beta/com/google/api/client/googleapis/json/GoogleJsonResponseException.html#getDetails()

Returns the Google JSON error details or null for none (for example if response is not JSON).

So basically, I'm catching a GoogleJsonResponseException but there is no JSON.

Why in the world can I catch a GoogleJsonResponseException when there's not actually any Json associated with it? Why didn't it instead catch HttpResponseException?

I want my app to be able to catch/use Json if it's returned, but I don't know how to do that reliably when GoogleJsonResponseException isn't even guaranteed to return Json.

Any suggestions?

Should I just be using HttpResponseException to reliably check statusCode and then looking at getMessage() to see if there's JSON?

Or should I just use GoogleJsonResponseException but call it's parent HttpResponseException method getStatusCode() to see status code, and only getDetails() if I really want to check/dive into any possible Json?

This exponential-backoff is using Drive API v2 Client lib(1.19.0) and is based off Google's current Java code samples at https://developers.google.com/drive/web/handle-errors I think Google needs to update this page... — Don Cheadle, Sep 02 '14 at 14:25
You really need to be catching a 403 to deal with flow control, and you need to trace the http to see what's happening on the wire. I suggest you post your question to https://groups.google.com/forum/#!forum/google-http-java-client — pinoyyid, Jan 29 '15 at 04:49
precisely what status codes I'm checking is irrelevant -- the question is focused around the `null` returned by `GoogleJsonResponseException.getDetails()`. Thanks for the google forum link... I'll try that. — Don Cheadle, Jan 29 '15 at 15:23
It's irrelevant to your question, but relevant to the robustness of your code in the face of Drive's behaviour. However, do feel free to throw an error instead of retrying. Your users will undoubtedly thank you for it. — pinoyyid, Jan 29 '15 at 18:02
I think at least SOME `exponential-retry`, for reasonable cases such as 5xx and 403, is expected / should have been built into the Drive SDK client (for example, AWS does this for you with its SDK), so I include a few retries before throwing it in my wrapper around the Drive client — Don Cheadle, Jan 29 '15 at 20:24
I agree. The problem exists because Google auto-generates the SDK based solely on the *spec* of the REST API/JSON, and does nothing to deal with the *behaviour* of the API. imho, if there is a flow control 403 triggered, then the 403 headers should include a transaction-id and a backoff-time. Our apps then know exactly how long to backoff for, and can resubmit the same attempt, which Drive can duplicate based on the tx-id. I believe Dropbox does at least part of this. — pinoyyid, Jan 30 '15 at 04:40
@pinoyyid you seem familiar with Drive, maybe with Java SDK? -- how do you handle `exponential-backoff-retry`? Could you post an answer to my Q? — Don Cheadle, Jan 30 '15 at 15:27
Hi. Your question is about error handling within the Google http Java client. In the answer I gave before are links to the source code. Having said that, all code has now been moved to Github, but the links below will lead you there. Your best route to an answer would be read the source code, or post your question at https://groups.google.com/forum/#!forum/google-http-java-client which is manned by the developers of the library. Even though you are trying to access Drive, your problem lives lower down the stack, so won't be seen by the correct Googlers by posting here on SO. hth — pinoyyid, Jan 30 '15 at 17:08
backoff-retry is a lottery with Drive. Some errors are hard, so all the backoff-retry in the world won't help, but destroys the user experience. Other errors are transient, and will succeed on a subsequent attempt. Flow control issues are best dealt with by avoiding them altogether by voluntary throttling. Unfortunately, there is little help from either the documentation or from the http responses, to help you determine which strategy works best. Ironically I once asked exactly the same question of Google and the answer was more-or-less, "dunno, you tell us". — pinoyyid, Jan 30 '15 at 17:12
... it took me this long to realize you're the one who originally responded — Don Cheadle, Jan 30 '15 at 21:57

score 2 · Answer 1 · edited May 23 '17 at 12:31

I'll post this as an answer so it's there for those in the same position, even though it doesn't address the specific questions you've asked.

A couple of suggestions

If you are uploading more than 30 files you will hit flow control issues. Drive's flow control is broken so you should throttle voluntarily rather than become a victim. See 403 rate limit on insert sometimes succeeds and 403 rate limit after only 1 insert per second
Enable http logging (see the code.google.com pages for the Java httpcliebnt for howto). This will at least allow you to see what the response is and maybe provide some clues on how best to trap the exception.
The page you linked to was for version 1.6. afaik the library is currently at 1.19 so check that you are using the latest lib and the latest documentation

What adds to the confusion is that there are three separate libraries.

At the bottom of the stack is the http library. https://code.google.com/p/google-http-java-client/
above that is the general api library. https://code.google.com/p/google-api-java-client/ and an oauth library https://code.google.com/p/google-oauth-java-client/
at the top are libraries for the various Google services that use the new API infrastructure. So eg. the Drive library is at https://developers.google.com/api-client-library/java/apis/drive/v2

Thanks for your response... The docs for the Java google drive api are honestly confusing to me. I can't figure out where the most valid docs are. Is is here: http://javadoc.google-api-java-client.googlecode.com/hg/1.6.0-beta/com/google/api/client/googleapis/json/GoogleJsonResponseException.html#getDetails() Or here: https://developers.google.com/resources/api-libraries/documentation/drive/v2/java/latest/ — Don Cheadle, Aug 31 '14 at 17:57
it seems there is **a lot** that is not clear-cut or well-documented with Google Drive SDK/Apps... Thanks for helping out with what you knew — Don Cheadle, Feb 03 '15 at 20:58

Catching GoogleJsonResponseException but e.getDetails() returns null

1 Answers1