How do I set an environment or substitution variable via a step in Google Cloud Build?

Question

Basically, when using Google Cloud Build, how do I read a value that was written in an earlier build step in subsequent steps?

Specifically, I'd like to make a custom image tag that's based on a combination of the timestamp and $SHORT_SHA. Something like the below. Though, it doesn't work, as docker complains about "export", and, even if that worked, it likely will be a different env:

  # Setting tag in a variable:
  - name: 'ubuntu'
    args: ['export', '_BUILD_TAG=`date', '-u', '+%Y%m%dT%H%M%S_$SHORT_SHA`']

Then, in a later step:

  # Using tag from the variable:
  - name: gcr.io/cloud-builders/docker
    args: ['build', '-t', 'gcr.io/$PROJECT_ID/$_BUILD_TAG', '.']

So, how do I use the output of one step in another? I could write the contents of date to a file, and then read it, but I'm back at not knowing how to set the variable from the file I read (or otherwise interpolate its results to form the argument to docker build).

score 23 · Accepted Answer · answered Sep 20 '18 at 01:11

I never found a way to set an environment variable in one build step that can be read in other steps, but I ended up accomplishing the same effect by building on Konstantin's answer in the following way:

In an early step, I generate and write my date-based tag to a file. The filesystem (/workspace) is retained between steps, and serves as store of my environment variable. Then, in each step that I need to reference that value, I cat that file in place. The trick is to use sh or bash as the entrypoint in each container so that the sub-shell that reads from the file can execute.

Here's an example:

## Set build tag and write to file _TAG
- name: 'ubuntu'
  args: ['bash', '-c', 'date -u +%Y%m%dT%H%M_$SHORT_SHA > _TAG']

...

# Using the _TAG during Docker build:
- name: gcr.io/cloud-builders/docker
entrypoint: sh
args: ['-c', 'docker build -t gcr.io/$PROJECT_ID/image_name:$(cat _TAG) .']

A caveat to note is that if you are doing the bash interpolation in this way within, say, a JSON object or something that requires double quotes, you need the subshell call to never be surrounded by single quotes when executed in the container, only double, which may require escaping the internal double quotes to build the JSON object. Here's an example where I patch the kubernetes config using the _TAG file value to deploy the newly-build image:

- name: gcr.io/cloud-builders/kubectl
entrypoint: bash
args: ['-c', 'gcloud container clusters get-credentials --zone $$CLOUDSDK_COMPUTE_ZONE $$CLOUDSDK_CONTAINER_CLUSTER ; kubectl patch deployment deployment_name -n mynamespace -p "{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\":\"image_name\",\"image\":\"gcr.io/$PROJECT_ID/image_name:$(cat _TAG)\"}]}}}}}"']
env:
- 'CLOUDSDK_COMPUTE_ZONE=us-central1-b'
- 'CLOUDSDK_CONTAINER_CLUSTER=my-google-proj-cluster-name'

Thanks @JJC. if you add a "/" (root dir) it works from all directories. Example: `echo "staging" > /_ENV && cd some_dir && echo "The value of \"_ENV\" is $(cat /_ENV)."`. Just remember to always surround `"$(cat /_ENV)"` with quotes (e.g. `mv /workspace/vars.auto.tfvars "environment/env/$(cat /_ENV)/vars.auto.tfvars"`) — Clemens, Mar 27 '20 at 17:09

score 4 · Answer 2 · answered Sep 17 '18 at 12:38

4

- name: gcr.io/cloud-builders/docker
  entrypoint: sh
  args
    - '-c'
    - 'docker build -t gcr.io/$PROJECT_ID/$(date -u +%Y%m%dT%H%M%S_$SHORT_SHA) .'

answered Sep 17 '18 at 12:38

Konstantin Tarkus

37,618
14
135
121

Thanks. This solves part of the issue. The remaining part of the question is that, since I want to re-use this tag in several other steps, I need it to be available there as well. If time passes between steps, I can't use the same shell invocation each time, as the tag value would change. So, I need to generate it early and then reference it in other steps. Any advice on that? – JJC Sep 17 '18 at 17:37
(1) either remove `T%H%M%S` portion from your version number (recommended), or (2) save the version number into a file, so you could reuse it in other steps: `echo $(date -u +%Y%m%dT%H%M%S_$SHORT_SHA) > ./VERSION` – Konstantin Tarkus Sep 20 '18 at 07:56
Yup, thanks! Before this comment I actually ended up using the latter approach, combined with reading that file in a subshell in subsequent steps. See my answer below. – JJC Sep 20 '18 at 14:25

score 1 · Answer 3 · answered Jan 23 '19 at 12:59

Although this doesn't solve your problem, I did want to post this answer since the first sentence of your question is, "Basically, when using Google Cloud Build, how do I read a value that was written in an earlier build step in subsequent steps?". This is how you'd do that.

From the official documentation:

A Volume is a Docker container that is mounted into build steps to persist files across build steps. When Cloud Build runs a build step, it automatically mounts a workspace volume into /workspace. You can specify additional volumes to be mounted into your build steps' containers using the volumes field for your steps.

Here's an example of it implemented from someone who asked this question in a github issue, but to put the date in the volume for later reading by another step:

steps:
- name: 'ubuntu'
  volumes:
  - name: 'vol1'
    path: '/persistent_volume'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        date -u +%Y%m%dT%H%M_$SHORT_SHA > /persistent_volume/file
- name: 'gcr.io/cloud-builders/docker'
  volumes:
  - name: 'vol1'
    path: '/persistent_volume'
  args: ['run', '-v', 'vol1:/data', 'alpine', 'cat', 'data/file']

However, for your particular case, I would just tag it with a subshell command like it's done in this answer here:

$(date -u +%Y%m%dT%H%M%S_$SHORT_SHA)

Thanks. Though, if you read the answer already given above, you'll see I'm basically doing just that, in the existing persistent environment/disk, without needing to mount an extra volume on each step. Is there an advantage to that, which I'm missing? — JJC, Jan 23 '19 at 20:33
For your particular case, there's no advantage. It'd probably be a net negative to include all that overhead where you could shell out inline instead. But, if you have someone coming here later who is looking to do this, but with, say, `ssh-keygen` or something with sizable output, using a volume to share data between steps might clean things up. — yurisich, Jan 24 '19 at 14:00

score 1 · Answer 4 · answered Nov 24 '19 at 19:29

Here's an example of what I've just done myself to reuse output from GitVersion in another. It builds on the answer @chetabahana posted.

steps:
- id: 'Gitversion: Unshallow repo'
  name: gcr.io/cloud-builders/git
  args: [fetch, --unshallow]

- id: 'Gitversion: Parse'
  name: gittools/gitversion:latest-linux
  entrypoint: /bin/bash
  args:
    - -c 
    - |
      dotnet /app/GitVersion.dll > /workspace/gitversion.json

- id: 'Gitversion: Env file'
  name: stedolan/jq
  entrypoint: /bin/bash
  args:
    - -c
    - |
      for s in $(cat /workspace/gitversion.json | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" ); do
        echo "export $s" >> /workspace/gitversion.env
      done

- id: 'Build and push API image'
  name: gcr.io/cloud-builders/docker
  entrypoint: /bin/bash
  args:
    - -c
    - |
      source /workspace/gitversion.env
      docker build -t gcr.io/xxxx/example:$${SemVer}-$${BuildMetaData} example-app
      docker push gcr.io/xxxx/example:$${SemVer}-$${BuildMetaData}

The magic ingredient was the $$ to escape substitution variable so that the build job would not attempt to substitute, leaving it for bash to substitute instead.

hi, quick question. what are we trying to imply with '-c' in args: -c ? — nanu146, Mar 07 '20 at 07:52
The `-c` is a command switch for bash to get it to interpret and execute the next string as a command. See https://askubuntu.com/questions/831847/what-is-the-sh-c-command — Pricey, Mar 08 '20 at 23:22

score 1 · Answer 5 · answered Mar 25 '20 at 10:20

Too bad this isn't supported (yet) by Google. However, I have been using the following simple method a lot and it works fine. Keep in mind that the file is saved to /workspace by default, which is shared among containers. If you would need it in a different directory, save or copy it somewhere else.

# Save variable to file
- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    _id=$(openssl rand -hex 16,,)
    echo "$${_id}" > id.txt

# Set variable from file
- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    _id=$(cat id.txt)
    echo "$${_id}"

Yes, this is what I ended up doing, if you read the top answer. Thanks. — JJC, Mar 25 '20 at 18:56

eQ19 · Answer 6 · 2019-07-08T17:33:59.213

You don't need to export nor mount a volume in your case.

steps:
- name: 'ubuntu'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        printenv

- name: gcr.io/cloud-builders/docker
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        printenv

It will output

BUILD
Starting Step #0
Step #0: Pulling image: ubuntu
Step #0: Using default tag: latest
Step #0: latest: Pulling from library/ubuntu
Step #0: Digest: sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395
Step #0: Status: Downloaded newer image for ubuntu:latest
Step #0: HOSTNAME=XXXXXXXXXXX
Step #0: BUILDER_OUTPUT=/builder/outputs
Step #0: PWD=/workspace
Step #0: HOME=/builder/home
Step #0: SHLVL=1
Step #0: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Step #0: _=/usr/bin/printenv
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: HOSTNAME=XXXXXXXXXXX
Step #1: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Step #1: PWD=/workspace
Step #1: SHLVL=1
Step #1: HOME=/builder/home
Step #1: DEBIAN_FRONTEND=noninteractive
Step #1: BUILDER_OUTPUT=/builder/outputs
Step #1: _=/usr/bin/printenv
Finished Step #1

So you can use /workspace or /builder/home, but since we cannot use a variable other than the defined substitution on yaml file then put them as script in the repo like this:

steps:
- name: 'ubuntu'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        bash test.bash

- name: gcr.io/cloud-builders/docker
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        bash result.bash

test.bash

#!/bin/bash
SHORT_SHA=myvar
date -u +%Y%m%dT%H%M_$SHORT_SHA > /workspace/myfile.txt

result.bash

#!/bin/bash
_BUILD_TAG=`cat /workspace/myfile.txt`
echo "the transferred value is: $_BUILD_TAG"

Output:

BUILD
Starting Step #0
Step #0: Pulling image: ubuntu
Step #0: Using default tag: latest
Step #0: latest: Pulling from library/ubuntu
Step #0: Digest: sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395
Step #0: Status: Downloaded newer image for ubuntu:latest
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: the transferred value is: 20190708T1706_myvar
Finished Step #1
PUSH
DONE

How do I set an environment or substitution variable via a step in Google Cloud Build?

6 Answers6

Linked