1

Here is a precise problem I'd like to solve programmatically (e.g. not manually doing some actions by hand in a browser).

The machine has Google Cloud SDK installed.

The gsutil ls command works successfully

However the python3 -c "from google.cloud import storage; storage.Client().list_buckets()" fails:

google.auth.exceptions.DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started

I need to do something programmatically to make google.cloud.storage work (using the credentials/auth mechanism used by gsutil). If gsutil can access the credentials, then google.cloud.storage should be able to do that as well.

How can I do that?

I probably need to set GOOGLE_APPLICATION_CREDENTIALS to the JSON file that gsutil is using, but what file does it use and are formats compatible?

Ark-kun
  • 6,358
  • 2
  • 34
  • 70
  • Could you check the following [documentation](https://cloud.google.com/storage/docs/reference/libraries#setting_up_authentication) and let me know if this works for you? – tzovourn Jun 08 '20 at 07:54

2 Answers2

1

You just need to add to the environment the variable GOOGLE_APPLICATION_CREDENTIALS and set it to your credentials

casellimarco
  • 109
  • 3
  • Can you please provide a programmatic way to fix the issue? – Ark-kun Jun 07 '20 at 09:11
  • Can't you source environment as part of your pipeline? Otherwise you can set the variable from python with `os.environ` https://stackoverflow.com/questions/5971312/how-to-set-environment-variables-in-python – casellimarco Jun 07 '20 at 09:26
  • I've mentioned the environment variable in my question. But your answer does not quite mention to which value the code needs to set it (programmatically). – Ark-kun Jun 07 '20 at 21:10
1

Initialize your client with credentials:

import os

from google.cloud import storage
from google.oauth2 import service_account

path_to_service_account_key = "key.json"  # TODO

credentials = service_account.Credentials.from_service_account_file(
        filename=path_to_service_account_key,
        scopes=['https://www.googleapis.com/auth/cloud-platform']
)
client = storage.Client(credentials=credentials)
client.list_buckets()

By default, gsutil uses the default service account for your project, which you can list with:

gcloud iam service-accounts list

If you want to use this same service account, or generate a new service account key entirely, you can do that from the IAM console: https://cloud.google.com/iam/docs/creating-managing-service-account-keys

Dustin Ingram
  • 20,502
  • 7
  • 59
  • 82
  • From the question: "I probably need to set GOOGLE_APPLICATION_CREDENTIALS to the JSON file that gsutil is using, but what file does it use and are formats compatible?" The "TODO" in your answer is what the question is all about. gsutil uses something. I need to use the same thing. – Ark-kun Jun 10 '20 at 18:51
  • It's not clear to me what you're asking. Are you asking how to generate that file programmatically? Or where that file comes from? See https://cloud.google.com/iam/docs/creating-managing-service-account-keys – Dustin Ingram Jun 10 '20 at 19:25
  • I've updated my answer with what I think you're asking for. – Dustin Ingram Jun 10 '20 at 19:30
  • >"you can do that from the IAM console" - This does not look like a programmatic way. See the very first line of the question: "Here is a precise problem I'd like to solve programmatically (e.g. not manually doing some actions by hand in a browser).'. Also, I do not think I want to create any new keys. I do not want to modify the user's project. gsutil already works somehow. I'd like to just use the same keys it uses. – Ark-kun Jun 10 '20 at 20:15