Setting an active gcloud account in docker container

Question

Currently i'm setting up a Kubeflow Pipeline on GKE. The goal is to start a trainingjob on the ML Engine and later on serve it on GKE.

The trainingjob gets launched in a Docker container. (Every step in a pipeline must be a container.)

I'm getting the following error when running the container:

ERROR: (gcloud.ml-engine.jobs.submit.training) You do not currently have an active account selected.
Please run:

  $ gcloud auth login

to obtain new credentials, or if you have already logged in with a
different account:

  $ gcloud config set account ACCOUNT

to select an already authenticated account to use.

The docker containers gets credentials via service account as suggested in following answer.

FROM tensorflow/tensorflow:1.8.0-devel-gpu-py3

RUN apt-get update -y && apt-get install --no-install-recommends -y -q ca-certificates python-dev python-setuptools wget unzip git


# Components to run ML Engine job on cluster
RUN cd / && \
    wget -nv https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.zip && \
    unzip -qq google-cloud-sdk.zip -d tools && \
    rm google-cloud-sdk.zip && \
    tools/google-cloud-sdk/install.sh --usage-reporting=false \
        --path-update=false --bash-completion=false \
        --disable-installation-options && \
    tools/google-cloud-sdk/bin/gcloud -q components update \
        gcloud core gsutil && \
    tools/google-cloud-sdk/bin/gcloud config set component_manager/disable_update_check true && \
    touch /tools/google-cloud-sdk/lib/third_party/google.py

ENV PATH $PATH:/tools/node/bin:/tools/google-cloud-sdk/bin

RUN mkdir /workdir

COPY . /workdir

RUN export GOOGLE_APPLICATION_CREDENTIALS=/workdir/ml6-sandbox-cdc8cb4bcae2.json

ENTRYPOINT ["bash", "/workdir/ml-engine/train.sh"]

The error finds place in train.sh where i'm submitting a trainingjob:

gcloud ml-engine jobs submit training $JOB_NAME \
    --job-dir $JOB_DIR \
    --runtime-version 1.8 \
    --python-version 3.5 \
    --module-name trainer.run_train \
    --package-path ./trainer \
    --region $REGION \
    --config=trainer/config.yaml \
    --stream-logs \
    -- \
    --data-dir $DATA_DIR \
    --version $VERSION

In my run_train.py, i'm getting the Google Application Credentials the following:

os.environ[
        "GOOGLE_APPLICATION_CREDENTIALS"] = '/workdir/ml6-sandbox-cdc8cb4bcae2.json'

Train.sh works standalone.

llompalles · Accepted Answer · 2019-03-07T10:36:09.383

You only need to set the GOOGLE_APPLICATION_CREDENTIALS env variable when using the client libraries.

As you are using the gcloud CLI change this line:

RUN export GOOGLE_APPLICATION_CREDENTIALS=/workdir/ml6-sandbox-cdc8cb4bcae2.json

to

gcloud auth activate-service-account yourServiceAccount --key-file=/workdir/ml6-sandbox-cdc8cb4bcae2.json

This will log your service account as the the active account used by gcloud.

Also, this service account needs to be granted with the proper roles.

Setting an active gcloud account in docker container

1 Answers1