Google Cloud storage on dev_appserver on Cloud Shell

Question

I am trying to use Google Cloud Storage from GAE and all works fine when deployed. I am however unable to get the storage to work on the development server that runs on Google Cloud Shell.

I am using the cloudstorage API and my understanding it that dev_appserver should access same buckets as the deployed application. For me working with local data would be fine as well.

The development server is invoked by dev_appserver.py .

I have played around with the non-documented default_gcs_bucket_name flag but no luck.

I have set he cloud shell to work with the correct project using gcloud init. I have also tried to use the new google.cloud.storage API without luck.

The same error as referenced below is triggered if I run the sample provided here: App Engine and Google Cloud Storage Sample.

Anyone?

Edit: I get the same error regardless of whether I supply a valid or an invalid bucket name.

app.yaml

runtime: python27
api_version: 1
threadsafe: true

builtins:
- remote_api: on

handlers:
- url: /.*
  script: main.app

main.py

import logging
import cloudstorage as gcs
import webapp2
from google.appengine.api import app_identity

class MainPage(webapp2.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        bucket_name = "xxxx-xxxxxx.appspot.com"     
        stats = gcs.listbucket('/'+bucket_name) 
        for stat in stats:
          self.response.write(repr(stat) + '')        

app = webapp2.WSGIApplication([
    ('/', MainPage),
], debug=True)

Log

INFO     2017-06-07 20:36:42,068 devappserver2.py:116] Skipping SDK update check.
INFO     2017-06-07 20:36:42,105 api_server.py:297] Starting API server at: http://0.0.0.0:38829
INFO     2017-06-07 20:36:42,110 dispatcher.py:209] Starting module "default" running at: http://0.0.0.0:8080
INFO     2017-06-07 20:36:42,111 admin_server.py:116] Starting admin server at: http://0.0.0.0:8000
ERROR    2017-06-07 20:36:54,836 api_server.py:374] Exception while handling service_name: "app_identity_service"
method: "GetAccessToken"
request: "\n7https://www.googleapis.com/auth/devstorage.full_control"
request_id: "QBrGUwjoJT"

Traceback (most recent call last):
  File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 349, in _handle_POST
    api_response = _execute_request(request).Encode()
  File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 225, in _execute_request
    make_request()
  File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/api_server.py", line 220, in make_request
    request_id)
  File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/api/apiproxy_stub.py", line 131, in MakeSyncCall
    method(request, response)
  File "/google/google-cloud-sdk/platform/google_appengine/google/appengine/api/app_identity/app_identity_defaultcredentialsbased_stub.py", line 195, in _Dynamic_GetAccessToken
    'expires': now + token.expires_in,
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

WARNING  2017-06-07 20:36:54,838 tasklets.py:468] suspended generator _make_token_async(rest_api.py:55) raised RuntimeError(TypeError("unsupported operand type(s) for +: 'int' and 'NoneType'",))
WARNING  2017-06-07 20:36:54,839 tasklets.py:468] suspended generator get_token_async(rest_api.py:224) raised RuntimeError(TypeError("unsupported operand type(s) for +: 'int' and 'NoneType'",))
WARNING  2017-06-07 20:36:54,839 tasklets.py:468] suspended generator urlfetch_async(rest_api.py:259) raised RuntimeError(TypeError("unsupported operand type(s) for +: 'int' and 'NoneType'",))
WARNING  2017-06-07 20:36:54,839 tasklets.py:468] suspended generator run(api_utils.py:164) raised RuntimeError(TypeError("unsupported operand type(s) for +: 'int' and 'NoneType'",))
WARNING  2017-06-07 20:36:54,839 tasklets.py:468] suspended generator do_request_async(rest_api.py:198) raised RuntimeError(TypeError("unsupported operand type(s) for +: 'int' and 'NoneType'",))
WARNING  2017-06-07 20:36:54,839 tasklets.py:468] suspended generator do_request_async(storage_api.py:137) raised RuntimeError(TypeError("unsupported operand type(s) for +: 'int' and 'NoneType'",))

Check your local account using `gcloud init`. You may not be using the same Google account to access the Storage from prod and local. — Rad Apdal, Jun 15 '17 at 01:45
Thanks! In cloud shell I am logged in as myslef and on GCE it is a service account. I am the owner of the project and have rights to the bucket so that should not be the problem. — Arne S, Jun 15 '17 at 06:59
Great. I'll add an answer that includes setting permissions. — Rad Apdal, Jun 15 '17 at 07:06
Hi Arne, Grace period ends in 3 hours, could you review my answer? Thanks — Mar Cial R, Jun 18 '17 at 07:11

Mar Cial R · Accepted Answer · 2017-06-17T08:41:15.070

1

Follow the instructions available in How the Application Default Credentials work (condition 1.)

It involves setting GOOGLE_APPLICATION_CREDENTIALS env variable to point to a valid json credentials file. For these tests, I just created a new one selecting App Engine default service account:

...and it worked beautifully.

According to How the Application Default Credentials work running gcloud auth application-default login (condition 2.) should also solve the issue but I have not been able to make it work on Cloud Shell, it fails with the same "access token related" error :-(

edited Jun 17 '17 at 08:41

answered Jun 17 '17 at 08:02

Mar Cial R

936
9
20

Hi Mar -thanks but I do not understand how this answer relates to my question. I am using Cloud Shell and are logged in as project owner. The dev_servers authorization is to my understanding taken care of by the cloud shell environment. – Arne S Jun 18 '17 at 07:32
Hi Arne, I cannot explain why it does not work from plain Cloud Shell. That also strikes me as some bug or unexplained feature. I'd still recommend using the higher precedence condition of setting the GOOGLE_APPLICATION_CREDENTIALS env variable. That way your code should work regardles of where you execute it.Give it a quick try and let me know if it solves your issue. – Mar Cial R Jun 18 '17 at 08:39
Thanks Mar - I did try and it made no difference – Arne S Jun 18 '17 at 08:44
I have not used on my tests your exact same code but the one you referenced on github (https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/storage/appengine-client/main.py) . I did start with your same error, then tried condition 2. and then setting env var solved it. I am now inclined to believe that either: 1 you are not setting the env var correctly, 2 there is something missing on your code ( like not using app_identity or a wrong gs bucket path) – Mar Cial R Jun 18 '17 at 08:51
I've just reverified everything. Try with the "appengine-client" code [1] as is and use: export GOOGLE_APPLICATION_CREDENTIALS=/home/.json [1] https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/storage/appengine-client/main.py – Mar Cial R Jun 18 '17 at 09:04
I am glad you've finally accepted my answer. Thanks. As a last request, it looks there is are a few minutes left to claim those 100 rep points... ;-) – Mar Cial R Jun 18 '17 at 09:22
Thank you for your persistence. I revisited you solution and using the steps provided I was able to get it working. What i did not understand is that it is now working by reading and saving to a "local" bucket (in cloud shell) and not the deployed bucket. As stated in question - that works for me. – Arne S Jun 18 '17 at 09:25

score 0 · Answer 2 · answered Jun 15 '17 at 07:11

0

dev_appserver.py uses configurations set in your local environment. Make sure the configs set on gcloud init is also correct for deploying to production.

To set Cloud Storage account permissions, go to Cloud Console then:

Storage > Browser > Right Button of your bucket > Edit bucket permissions

Add your account as Storage Admin or the likes.

answered Jun 15 '17 at 07:11

Rad Apdal

442
1
6
16

Thanks again, but as stated in comment above I run dev_appserver using the account that is the owner of the project and I have verified that his user has full rights to the bucket. – Arne S Jun 15 '17 at 08:03

score 0 · Answer 3 · answered Oct 21 '20 at 09:36

0

Make sure you're logged in and have authorized the default service credentials.

% gcloud auth login
% gcloud auth application-default login

answered Oct 21 '20 at 09:36

Manfred Georg

1

Google Cloud storage on dev_appserver on Cloud Shell

3 Answers3