6

This post is a followup to How to make 'access_type=offline' / server-only OAuth2 operations on GAE/Python. The http = credentials.authorize(httplib2.Http()) part no longer fails when testing, but it seems it still does when run by GAE's cron, where it's unable to refresh my access_token :

  1. I can manually run my job by calling /fetch, say at 11:45.
  2. Scheduling immediately a /cronfetch job at 11:55 works then, without any access_token issue.

  3. But then, I woke up this morning seeing that the same /cronfetch task (same except the timing, which is at 01:00 for my non-test daily task) failed:

    I 2013-06-10 05:53:51.324 make: Got type <class 'google.appengine.api.datastore_types.Blob'>
I 2013-06-10 05:53:51.325 validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
I 2013-06-10 05:53:51.327 URL being requested: https://www.googleapis.com/youtube/v3/playlists?alt=json&part=snippet%2Cstatus
I 2013-06-10 05:53:51.397 Refreshing due to a 401
I 2013-06-10 05:53:51.420 make: Got type <class 'google.appengine.api.datastore_types.Blob'>
I 2013-06-10 05:53:51.421 validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
I 2013-06-10 05:53:51.421 Refreshing access_token
I 2013-06-10 05:53:51.458 Failed to retrieve access token: { "error" : "invalid_grant" }
I 2013-06-10 05:53:51.468 make: Got type <class 'google.appengine.api.datastore_types.Blob'>
I 2013-06-10 05:53:51.468 validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
I 2013-06-10 05:53:51.471 validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
I 2013-06-10 05:53:51.471 get: Got type <class 'oauth2client.appengine.CredentialsModel'>
E 2013-06-10 05:53:51.480 invalid_grant Traceback (most recent call last): File "/python27_runtime/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in

This Getting "invalid_grant" error on token refresh mailing list message (+ SO post 1, SO post 2, SO post 3) look similar to my problem, but it seems to be happening with a access_type=online token. In my case I just use the default access_type=offline, and I see the "Perform these operations when I'm not using the application" mention in the initial access request.

I just re-scheduled a cron run at 08:25 (taking care not to launch a manual one) with debug print statements that I committed to GitHub for you. Here is what I get, it is similar but not identical (Note the few last lines seem ordered incorrectly, I definitely am not doing OAuth2 stuff in create_playlist until all sources are read). So ignoring the skewed order (GAE logging artifact?), it seems my http = credentials.authorize(Http()) call in create_playlist(self), currently at line 144 is wrong:

    ...
    E 2013-06-10 08:26:12.817 http://www.onedayonemusic.com/page/2/ : found embeds ['80wWl_s-HuQ', 'kb1Nu75l1vA', 'kb1Nu75l1vA', 'RTWcNRQtkwE', 'RTWcNRQtkwE', 'ZtDXezAhes8', 'ZtDXezAhes8', 'cFGxNJhKK9c', 'cFGxNJhKK9c'
    I 2013-06-10 08:26:14.019 make: Got type <class 'google.appengine.api.datastore_types.Blob'>
    I 2013-06-10 08:26:14.020 validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
    I 2013-06-10 08:26:14.022 URL being requested: https://www.googleapis.com/youtube/v3/playlists?alt=json&part=snippet%2Cstatus
    I 2013-06-10 08:26:14.100 Refreshing due to a 401
    I 2013-06-10 08:26:14.105 make: Got type <class 'google.appengine.api.datastore_types.Blob'>
    I 2013-06-10 08:26:14.106 validate: Got type <class 'oauth2client.client.OAuth2Credentials'>
    I 2013-06-10 08:26:14.106 Refreshing access_token
    E 2013-06-10 08:26:18.994 Deadline exceeded while waiting for HTTP response from URL: https://accounts.google.com/o/oauth2/token Traceback (most recent call last): File "/pyt
    E 2013-06-10 08:26:18.996 http://www.onedayonemusic.com/page/3/ : found embeds ['80wWl_s-HuQ', '6VNu2MLdE0c', '6VNu2MLdE0c', 'YwQilKbK9Mk', 'YwQilKbK9Mk', 'KYdB3rectmc', 'KYdB3
    E 2013-06-10 08:26:18.996 crawl_videos end
    E 2013-06-10 08:26:18.996 create_playlist start
    E 2013-06-10 08:26:18.996 create_playlist got creds
    E 2013-06-10 08:26:18.996 create_playlist authorized creds

→ Why does the cron job work 5min after a manual run, but fails 6hours later? I thought the refresh token never expired. What am I doing wrong?

Note that's my first GAE work, and my second Python program at all, general code review/advice is very welcome but please be gentle :)

The code is on GitHub and my instance can be reached at dailygrooves.org. Thanks for your help!

Community
  • 1
  • 1
Ronan Jouchet
  • 1,303
  • 1
  • 15
  • 28

1 Answers1

4

An invalid_grant is returned when the refresh token can't be used to get a new access token from the current user. This is happening to you because the stored Credentials object has a null refresh token, i.e.

>>> credentials.refresh_token is None
True

As mentioned in the NOTE in How to make 'access_type=offline' / server-only OAuth2 operations on GAE/Python?:

If a user has already authorized your client ID, the subsequent times you perform OAuth for these users they will not see the OAuth dialog and you won't be given a refresh token.

You need to make sure your Credentials are stored with a valid refresh token and the easiest way to do this, as mentioned in your last question as well as in all 3 questions you linked to is to use approval_prompt=force when creating your OAuth2WebServerFlow or OAuth2Decorator object (whichever you are using).

Community
  • 1
  • 1
bossylobster
  • 9,993
  • 1
  • 42
  • 61
  • OK well, I thought I didn't need that, because my cron task worked once! How can the same cron task work a few minutes after a manual task, but fail 6 hours later? And a 2nd question: since I'm using `OAuth2DecoratorFromClientSecrets`, which doesn't forward additional `**kwargs` to `OAuth2Decorator`, how do you recommend setting my additional `approval_prompt=force` parameter? Does it have to be done at init time, or is it OK if I create my `decorator = OAuth2DecoratorFromClientSecrets...` and do `decorator.params.update({approval_prompt='force'})` afterwards? Thanks. – Ronan Jouchet Jun 10 '13 at 18:23
  • 1
    The cron task will work when `credentials.access_token` is valid, but the access token will expire after 1 hour. After it expires, `credentials.refresh_token` is needed to get a new access token. You would want `decorator.params.update({'approval_prompt': 'force'})` but there is no good reason not to just pass it to the constructor. – bossylobster Jun 10 '13 at 19:21
  • OK, didn't know about the 1hour limit, thanks. Regarding *"there is no good reason not to just pass it to the constructor"*, well I would like to! But `OAuth2DecoratorFromClientSecrets` does not seem to honor/forward additional `**kwargs` (that's what I understand from the source and the exception it throws if I try), thus my question. Am I missing something? – Ronan Jouchet Jun 10 '13 at 19:39
  • Yes I can't comment on using `OAuth2DecoratorFromClientSecrets` as I've not used it but if you just used `OAuth2Decorator` you'd have no issue. Pick your poison on that one. – bossylobster Jun 10 '13 at 20:41
  • I still get a `invalid_grant` error, even after [this commit](https://github.com/ronjouch/dailygrooves/commit/0a8d2b5cce38dde487fddd6c609abe576f0cc977). My test scenario: 1. manual test (`/fetch`, which worked), 2. 70minutes of wait, 3. cron run (`/cronfetch`, which returns the `invalid_grant` error). Any idea? – Ronan Jouchet Jun 10 '13 at 22:10
  • You need to update the `DECORATOR` before it is used, by doing it in the method, you are updating it after and nothing occurs. If you don't see a new prompt, it will just use the credentials you have in the datastore. – bossylobster Jun 10 '13 at 22:13
  • I'd like to, but before invoking the decorator, its `flow` member is uninitialized, so my `decorator.flow.params.update({'approval_prompt': 'force'})` results in a `NoneType` exception. OK then, I'll try abandoning `OAuth2DecoratorFromClientSecrets` and just use `OAuth2Decorator`. – Ronan Jouchet Jun 10 '13 at 22:17
  • Hi @bossylobster. I am trying to test what I mentioned above, but am constantly getting `HTTPException: Deadline exceeded while waiting for HTTP response from URL: https://accounts.google.com/o/oauth2/token`, like [this SO post](http://stackoverflow.com/questions/16993204/google-app-engine-drive-sdk-catching-a-lot-http-deadline-exceptions) and [that mailing-list post](https://groups.google.com/forum/?fromgroups#!topic/google-appengine/BCdXSpDDYWc). Are we doing something wrong, or is it really a Google-side issue? Any workaround? – Ronan Jouchet Jun 11 '13 at 14:26
  • This is a Google-side issue, we are working on it. If you continue to have issues, please bring them up elsewhere. This question has been answered correctly. – bossylobster Jun 11 '13 at 16:36
  • Tested and working fine! My app is now up at [www.dailygrooves.org](http://www.dailygrooves.org/), I'll give you a questions break for a few days :D . A million thanks for all the help from you and the other people on `google-api-python-client`, I couldn't have done it without such great support! – Ronan Jouchet Jun 14 '13 at 15:27