How to provide Vault secrets for a Flink application custom resource in Kubernetes

Question

I would like to provide secrets from a Hashicorp Vault for the Apache Flink jobs running in a Kubernetes cluster. These credits will be used to access a state-backend for checkpointing and savepoints. The state-backend could be for example Minio S3 storage. Could someone provide a working example for a FlinkApplication operator please given the following setup?

Vault secrets for username and password (or an access key):

vault kv put vvp/storage/config username=user password=secret
vault kv put vvp/storage/config access-key=minio secret-key=minio123

k8s manifest of the Flink application custom resource:

apiVersion: flink.k8s.io/v1beta1
kind: FlinkApplication
metadata:
  name: processor
  namespace: default
spec:
  image: stream-processor:0.1.0
  deleteMode: None

  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: vvp-flink-job
        vault.hashicorp.com/agent-inject-secret-storage-config.txt: vvp/data/storage/config

  flinkConfig:
    taskmanager.memory.flink.size: 1024mb
    taskmanager.heap.size: 200
    taskmanager.network.memory.fraction: 0.1
    taskmanager.network.memory.min: 10mb
    web.upload.dir: /opt/flink
  jobManagerConfig:
    resources:
      requests:
        memory: "1280Mi"
        cpu: "0.1"
    replicas: 1
  taskManagerConfig:
    taskSlots: 2
    resources:
      requests:
        memory: "1280Mi"
        cpu: "0.1"
  flinkVersion: "1.14.2"
  jarName: "stream-processor-1.0-SNAPSHOT.jar"
  parallelism: 3
  entryClass: "org.StreamingJob"
  programArgs: >
    --name value

Docker file of the flink application:

FROM maven:3.8.4-jdk-11 AS build
ARG revision

WORKDIR /
COPY    src /src
COPY    pom.xml /

RUN mvn -B -Drevision=${revision} package

# runtime
FROM flink:1.14.2-scala_2.12-java11

ENV FLINK_HOME=/opt/flink

ENTRYPOINT ["/docker-entrypoint.sh"]
EXPOSE 6123 8081
CMD ["help"]

The flink-config.yaml contains the following examples:

# state.backend: filesystem

# Directory for checkpoints filesystem, when using any of the default bundled
# state backends.
#
# state.checkpoints.dir: hdfs://namenode-host:port/flink-checkpoints

# Default target directory for savepoints, optional.
#
# state.savepoints.dir: hdfs://namenode-host:port/flink-savepoints

The end goal is to replace the hardcoded secrets or set them somehow from the vault:

state.backend: filesystem
s3.endpoint: http://minio:9000
s3.path.style.access: true
s3.access-key: minio
s3.secret-key: minio123

Thank you.

what's the issue you have already added the annotation for the vault, if CRD is creating the deployment or POD your annotation will be there inside the POD or deployment and you will be getting the variable in file. https://learn.hashicorp.com/tutorials/vault/kubernetes-sidecar?in=vault/kubernetes#inject-secrets-into-the-pod — Harsh Manvar, Jan 05 '22 at 18:38
Hi @Harsh Manvar the issue is how to provide the variables from the Vault file for the Flink application itself. Should it be appended to the flink-config.yaml with a shell script or set via `StreamExecutionEnvironment getExecutionEnvironment(Configuration configuration)` during runtime from a Java/Scala application? `configuration.setString("s3.access-key", accessKey)`. I probably already answered my question to some extent with this comment. — Kubus, Jan 07 '22 at 07:15
you mean at end from the vault to your pod application should be able to access varible using environment ? right? vault injecting it as file inside pod but you want ti as environment varible so application get it from env instead parsing is it? — Harsh Manvar, Jan 07 '22 at 07:18
Actually that's yet another option to set it as an env. variable, isn't it? I was thinking about parsing the vault secrets provided in the pod's secret-storage-config.txt from my Scala application. — Kubus, Jan 07 '22 at 07:32
Yes, i will write answer instead of parsing getting variables from env would be easy still it depends on your development plan. you can inject this txt file lines as an environment variable or else parse the file is option. — Harsh Manvar, Jan 07 '22 at 07:33
Thank you very much @Harsh Manva! It would be nice if you can write down several different approaches, it will be valuable for other developers, too. — Kubus, Jan 07 '22 at 07:38

Harsh Manvar · Accepted Answer · 2022-01-07T07:53:40.827

Once you have vault variables set

You can add the annotation in deployment to get variables out of the vault to deployment

annotations:
        vault.hashicorp.com/agent-image: <Agent image>
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-secrets: kv/<Path-of-secret>
        vault.hashicorp.com/agent-inject-template-secrets: |2

          {{- with secret "kv/<Path-of-secret>" -}}

          #!/bin/sh
          set -e

          {{- range $key, $value := .Data.data }}
          export {{ $key }}={{ $value }}
          {{- end }}

          exec "$@"
          {{- end }}
        vault.hashicorp.com/auth-path: auth/<K8s cluster for auth>
        vault.hashicorp.com/role: app

this will create the file inside your POD.

When you application run it should execute this file first and the environment variable will get injected to POD.

So vault annotation will create one file the same as you are getting as txt but instead, we will be doing it like

{{- range $key, $value := .Data.data }}
       export {{ $key }}={{ $value }}
{{- end }}

it will keep and inject the export before key & value. Now the file is a kind of shell script and once it will get executed on the startup of the application it will inject variables to the OS level.

Keep this file in reop and add it in Docker ./bin/runapp

#!/bin/bash
if [ -f '/vault/secrets/secrets' ]; then
  source '/vault/secrets/secrets'
fi
node <path-insnide-docker>/index.js #Sorry dont know scala or Java

package.json

"start": "./bin/runapp",

Dockerfile

ADD ./bin/runapp ./
EXPOSE 4444
CMD ["npm", "start"]

Your vault injected file will be something like inside pod at /vault/secrets/secrets or your configured path.

#!/bin/sh
set -e
export development=false
export production=true
exec "$@"

How to provide Vault secrets for a Flink application custom resource in Kubernetes

1 Answers1

Linked