1

I want to access datastore (and storage) data of an AppEngine project via google-cloud-datastore and google-cloud-storage with an Python program on my own server.

This works with my AppEngine staging server, by creating a service account and giving it owner access (to the project).

Doing the same thing with the production AppEngine instance fails with

google.api_core.exceptions.PermissionDenied: 403 Missing or insufficient permissions.

Part of the problem might be, that I might be using the wrong project to create the service account with. There are more than one project with the same name in my cloud console. How do I identify the correct one?

How do I get more details about the problem?

Ruediger Jungbeck
  • 2,836
  • 5
  • 36
  • 59

1 Answers1

1

First, note that the Datastore and the Cloud Storage are 2 different products with 2 different accessing methods.

The Datastore is closely tied to the GAE project - each project has its own datastore. The external access procedure in general is captured in How do I use Google datastore for my web app which is NOT hosted in google app engine?.

When switching the project (staging to production in your case) there are 2 things to keep in mind:

  • as you observed, you need to change the project you're accessing.

  • you also need to change the credentials you load and use for access to match the project you select, as each project has it own service account key configured in the above-mentioned procedure

For the google-cloud-datastore library both of these are simultaneously configured via the datastore.Client() call parameters (emphasis mine):

class google.cloud.datastore.client.Client(project=None, namespace=None, credentials=None, _http=None, _use_grpc=None)

  • project (str) – (Optional) The project to pass to proxied API methods.
  • credentials (Credentials) – (Optional) The OAuth2 Credentials to use for this client. If not passed (and if no _http object is passed), falls back to the default inferred from the environment.

The Cloud Storage is completely independent from GAE, the GAE project/credentials you use (if any) have no bearing on bucket/object access restrictions whatsoever. There's nothing you need to do from the google-cloud-storage library perspective when switching from one GAE project to another

To eliminate the confusion created by multiple projects having the same name just go to the IAM & admin Settings page, select the respective projects from the drop-down list on the top blue bar and rename them using meaningful names (click in the Project name box to edit the name, then click SAVE). Then re-check if you're using the right keys for the desired project.

Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
  • I know that datastore and cloud storage are two different products, but I store references to blobs (in cloud storage) in the datastore. I used the GOOGLE_APPLICATION_CREDENTIALS to set the key file. As I said in my post, my code (and key files) works on the staging server (an own AppEngine project). So my problem is has probably something to do with the keyfile for the production project. One problem is that I have several projects with the same name (for whatever reason). I tried the keyfile of the AppEngine project that I use to see the logs, but had no success. – Ruediger Jungbeck Feb 11 '19 at 15:40
  • I think you need the project ID (which is unique), not the name: https://console.cloud.google.com/iam-admin/settings – Dan Cornilescu Feb 11 '19 at 16:04
  • I took the project id from the AppEngine dashboard, created an own service account for that project ID, create a JSON key, and gave it with IAM the project owner role. (That is the same that I did for the staging project where it works). The problem is, that I get Permission denied. – Ruediger Jungbeck Feb 11 '19 at 16:12
  • Actually the Google Cloud Storage API works with the key, but the Google Datastore access does not. – Ruediger Jungbeck Feb 12 '19 at 13:35
  • I understand this concept , but I still dont know which project is doing what (and if I still need all of them). Same problem with the countless service accounts and keys – Ruediger Jungbeck Feb 12 '19 at 22:43