How to inject secret from Google Secret Manager into Kubernetes Pod as environment variable with Spring Boot?

Question

For the life of Bryan, how do I do this?

Terraform is used to create an SQL Server instance in GCP. Root password and user passwords are randomly generated, then put into the Google Secret Manager. The DB's IP is exposed via private DNS zone.

How can I now get the username and password to access the DB into my K8s cluster? Running a Spring Boot app here.

This was one option I thought of:

In my deployment I add an initContainer:

- name: secrets
  image: gcr.io/google.com/cloudsdktool/cloud-sdk
  args: 
  - echo "DB_PASSWORD=$(gcloud secrets versions access latest --secret=\"$NAME_OF_SECRET\")" >> super_secret.env

Okay, what now? How do I get it into my application container from here?

There are also options like bitnami/sealed-secrets, which I don't like since the setup is using Terraform already and saving the secrets in GCP. When using sealed-secrets I could skip using the secrets manager. Same with Vault IMO.

You'd need to use a tool like Berglas: https://github.com/GoogleCloudPlatform/berglas — sethvargo, Sep 16 '20 at 15:55
Do you generate the password and put it into secret manager in Terraform? — Anton Daneyko, Jan 27 '22 at 12:50

score 5 · Answer 1 · answered Sep 17 '20 at 08:24

On top of the other answers and suggestion in the comments I would like to suggest two tools that you might find interesting.

First one is secret-init:

secrets-init is a minimalistic init system designed to run as PID 1 inside container environments and it`s integrated with multiple secrets manager services, e.x. Google Secret Manager

Second one is kube-secrets-init:

The kube-secrets-init is a Kubernetes mutating admission webhook, that mutates any K8s Pod that is using specially prefixed environment variables, directly or from Kubernetes as Secret or ConfigMap.

It`s also support integration with Google Secret Manager:

User can put Google secret name (prefixed with gcp:secretmanager:) as environment variable value. The secrets-init will resolve any environment value, using specified name, to referenced secret value.

Here`s a good article about how it works.

Neither of these 2 systems worked for me in my app. I tried them both. They both errored out and there were no errors specified in the logs. — Riley Hun, Mar 28 '21 at 01:00
I ended up using https://github.com/external-secrets/kubernetes-external-secrets. — Moritz Schmitz v. Hülst, Apr 29 '21 at 13:03

Arghya Sadhu · Answer 2 · 2020-09-16T16:09:05.890

4

How do I get it into my application container from here?

You could use a volume to store the secret and mount the same volume in both init container and main container to share the secret with the main container from the init container.

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: my-app
    image: my-app:latest
    volumeMounts:
    - name: config-data
      mountPath: /data
  initContainers:
  - name: secrets
    image: gcr.io/google.com/cloudsdktool/cloud-sdk
    args: 
    - echo "DB_PASSWORD=$(gcloud secrets versions access latest --secret=\"$NAME_OF_SECRET\")" >> super_secret.env
    volumeMounts:
    - name: config-data
      mountPath: /data
  volumes:
  - name: config-data
    emptyDir: {}

edited Sep 16 '20 at 16:09

answered Sep 16 '20 at 16:03

Arghya Sadhu

41,002
9
78
107

How do I read it back into the Spring Boot app though? – Moritz Schmitz v. Hülst Sep 17 '20 at 08:58
I have not tried this in practice but you can read it from File system using `spring.cloud.kubernetes.config.paths` or implement a PropertySource – Arghya Sadhu Sep 17 '20 at 09:08

score 1 · Answer 3 · answered Sep 16 '20 at 16:13

1

Using volumes of emptyDir with medium: Memory to guarantee that the secret will not be persisted.

...
volumes:
      - name: scratch
        emptyDir:
          medium: Memory
          sizeLimit: "1Gi"
...

answered Sep 16 '20 at 16:13

Chayne P. S.

1,558
12
17

How to read it back into env vars on Spring Boot side though? – Moritz Schmitz v. Hülst Sep 17 '20 at 08:57
Inspired by https://stackoverflow.com/a/20909045/3849555, you have to read the file before you start spring-boot. So in cmd should have this line before `java -jar xxx.jar`: `export $(grep -v '^#' /blah/blah/super-secrets.env | xargs -d '\n')` then the rest read the variable as how spring-boot do. – Chayne P. S. Sep 17 '20 at 14:05

score 1 · Accepted Answer · answered Apr 29 '21 at 09:56

1

You can use spring-cloud-gcp-starter-secretmanager to load secrets from Spring application itself.

Documentation - https://cloud.spring.io/spring-cloud-gcp/reference/html/#secret-manager

answered Apr 29 '21 at 09:56

ivanka

171
2
4

Have you tried this? Does this work with Workload Identities? Thanks! – Moritz Schmitz v. Hülst Apr 29 '21 at 12:58
1

I use it with service accounts, not workload identities. I have terraform code, that creates service account, key for it and stores key as secret in k8s, then application uses that secret to configure any of spring-cloud-gcp projects – ivanka Apr 30 '21 at 13:07

score 0 · Answer 5 · answered Jan 27 '22 at 18:14

If one has control over the image, it's possible to change the entry point and use berglas.

Dockerfile:

FROM adoptopenjdk/openjdk8:jdk8u242-b08-ubuntu  # or whatever you need

# Install berglas, see https://github.com/GoogleCloudPlatform/berglas
RUN mkdir -p /usr/local/bin/
ADD https://storage.googleapis.com/berglas/main/linux_amd64/berglas /usr/local/bin/berglas
RUN chmod +x /usr/local/bin/berglas

ENTRYPOINT ["/usr/local/bin/berglas", "exec", "--"]

Now we build the container and test it:

docker build -t image-with-berglas-and-your-app .
docker run \
    -v /host/path/to/credentials_dir:/root/credentials \
    --env GOOGLE_APPLICATION_CREDENTIALS=/root/credentials/your-service-account-that-can-access-the-secret.json \
    --env SECRET_TO_RESOLVE=sm://your-google-project/your-secret  \
    -ti image-with-berglas-and-your-app env

This should print the environment variables with the sm:// substituted by the actual secret value.

In K8s we run it with Workload Identity, so the K8s service account on behalf of which the pod is scheduled needs to be bound to a Google service account that has the right to access the secret.

In the end your pod description would be something like this:

apiVersion: v1
kind: Pod
metadata:
  name: your-app
spec:
  containers:
  - name: your-app
    image: image-with-berglas-and-your-app
    command: [start-sql-server]
    env:
      - name: AXIOMA_PASSWORD
        value: sm://your-google-project/your-secret

How to inject secret from Google Secret Manager into Kubernetes Pod as environment variable with Spring Boot?

5 Answers5

Linked