How to authenticate google APIs with different service account credentials?

Question

As anyone who has ever had the misfortune of having to interact with the panoply of Google CLI binaries programmatically will have realised, authenticating with the likes of gcloud, gsutil, bq, etc. is far from intuitive or trivial, especially when you need to work across different projects.

I am running various cron jobs that interact with Google Cloud Storage and BigQuery for different projects. Since the cron jobs may overlap, renaming config files is clearly not an option, and nor would any sane person take that approach.

There must surely be some sort of method of passing a path to a service account's key pair file to these CLI binaries, but bq help yields nothing.

The Google documentation, while verbose, is largely useless, taking one on a tour of how OAuth2 works, etc, instead of explaining what must surely be a very common requirement, vis-a-vis, how to actually authenticate a service account without running commands that modify central config files.

Can any enlightened being tell me whether the engineers at Google decided to add a feature as simple as passing the path to a service account's key pair file to the likes of gsutil and bq? Or perhaps I could simply export some variable so they know which key pair file to use for authentication?

I realise these simplistic approaches may be an insult to the intelligence, but we aren't concerning ourselves with harnessing nuclear fusion, so we needn't even consider what Amazon got so right with their approach to authentication in comparison...

Answer 1

Configuration in the Cloud SDK is global for the user, but you can specify what aspects of that config to use on a per command basis. To accomplish what you are trying to do you can:

gcloud auth activate-service-account foo@developer.gserviceaccount.com --key-file ...
gcloud auth activate-service-account bar@developer.gserviceaccount.com --key-file ...

At this point, both sets of credentials are in your global credentials store. Now you can run:

gcloud --account foo@developer.gserviceaccount.com some-command
gcloud --account bar@developer.gserviceaccount.com some-command

in parallel, and each will use the given account without interfering.

---

A larger extension of this is 'configurations' which do the same thing, but for your entire set of config (including settings like account and project).

# Create first configuration
gcloud config configurations create myconfig
gcloud config configurations activate myconfig
gcloud config set account foo@developer.gserviceaccount.com
gcloud config set project foo

# Create second configuration
gcloud config configurations create anotherconfig
gcloud config configurations activate anotherconfig
gcloud config set account bar@developer.gserviceaccount.com
gcloud config set project bar

And you can say which configuration to use on a per command basis.

gcloud --configuration myconfig some-command
gcloud --configuration anotherconfig some-command

You can read more about configurations by running: gcloud topic configurations

---

All properties have corresponding environment variables that allow you to set that particular property for a single command invocation or for a terminal session. They take the form:

CLOUDSDK_<SECTION>_<PROPERTY>

for example: CLOUDSDK_CORE_ACCOUNT

You can see all the available config settings by running: gcloud help config

The equivalent of the --configuration flag is: CLOUDSDK_ACTIVE_CONFIG_NAME

---

If you really want complete isolation, you can also change the Cloud SDK's config directory by setting CLOUDSDK_CONFIG to a directory of your choosing. Note that if you do this, the config is completely separate including the credential store, all configurations, logs, etc.