How to log into Google Cloud Storage from a python function?

Question

I am new to google cloud storage and I try to set up a function that downloads a blob once a day. At the moment I am working in my Jupyter Notebook but finally, the code will run in an Azure Function. I am struggling with setting up the client that connects me to the bucket. I have a service account credential JSON which enables me to connect to google.

Locally I have found a solution:

from google.cloud import storage

client = storage.Client.from_service_account_json('<PATH_TO_SERVICE_ACCOUNT_JSON>')

The problem is that I do not have a path where I store my JSON in the cloud but I store it in the key vault. I came up with the following solution:

from google.cloud import storage
import json
from google.oauth2 import service_account

string_key = get_key_from_key_vault()
service_account_info = json.loads(string_key)
google_credentials = service_account.Credentials.from_service_account_info(
    service_account_info
)
scoped_credentials = google_credentials.with_scopes(
    ['https://www.googleapis.com/auth/cloud-platform.read-only'])
print(type(scoped_credentials))
client = storage.Client(credentials = scoped_credentials)

I am not totally sure if I need the scoped_credentials = ...part but I only have read permissions on the bucket. (if I skip the part the error stays the same)

When I go for this solution I get the following error:

DefaultCredentialsError: Could not automatically determine credentials. Please set 
GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For
 more information, please see https://cloud.google.com/docs/authentication/getting-started

I do not have a clue what I am doing wrong because I think that I already set the credentials explicitly.

Best P

It's the correct way to do,I don't know why it doesn't work. I also don't know what is this `path_to_key`. Of you have a path, you can directly use it in the storage client. — guillaume blaquiere, Aug 24 '20 at 14:56
@guillaumeblaquiere, thanks for your answer. Thanks for the comment. There was a copy paste error in the code. I edited it seconds ago. — Pet, Aug 24 '20 at 15:04

score 1 · Answer 1 · answered Aug 24 '20 at 20:44

1

you can set the environment variable GOOGLE_APPLICATION_CREDENTIALS with the path of the json file and authenticate your function by starting the storage client without parameters.

client = storage.Client()

*by default the storage client uses the file path on the environment variable GOOGLE_APPLICATION_CREDENTIALS

It is the easiest way to use JSON credentials and it is compatible with most of Google Cloud python libraries.

answered Aug 24 '20 at 20:44

Jan Hernandez

4,414
2
12
18

Hello thanks for your answer, I totally understand your answer but I am looking for a solution where I do not have to store my JSON file in the storage. I want to read it from a key vault and this is where my problem starts :-( Best, P – Pet Aug 25 '20 at 05:20

score 1 · Accepted Answer · answered Aug 25 '20 at 08:41

after some more tests i found out that I missed to add project = None. If you add it an use the following command to create the client it works:

client = storage.Client(project = None, credentials = scoped_credentials)

Thanks for your help and food for thought :-)

score 0 · Answer 3 · answered Aug 24 '20 at 19:19

(I use the answer part because code formatting in comment is awful)

Can you try this and tell me if you see the 2 access token printed?

from google.cloud import storage
import json
from google.oauth2 import service_account
from google.auth.transport import requests as grequests

string_key = get_key_from_key_vault()
service_account_info = json.loads(string_key)

google_credentials = service_account.Credentials.from_service_account_info(
    service_account_info
)
google_credentials.refresh(grequests.Request())
print(google_credentials.token)


scoped_credentials = google_credentials.with_scopes(
    ['https://www.googleapis.com/auth/cloud-platform.read-only'])
scoped_credentials.refresh(grequests.Request())
print(scoped_credentials.token)

thanks for your answer. I tested your code and I can only see the second token (with scope). The first rises an error: ```RefreshError: ('invalid_scope: Invalid OAuth scope or ID token audience provided.', '{"error":"invalid_scope","error_description":"Invalid OAuth scope or ID token audience provided."}')``` — Pet, Aug 25 '20 at 05:14

How to log into Google Cloud Storage from a python function?

3 Answers3

Linked