Adjusting Kubernetes configurations depending on environment

Question

I want to describe my services in kubernetes template files. Is it possible to parameterise values like number or replicas, so that I can set this at deploy time.

The goal here is to be able to run my services locally in minikube (where I'll only need one replica) and have them be as close to those running in staging/live as possible.

I'd like to be able to change the number of replicas, use locally mounted volumes and make other minor changes, without having to write a seperate template files that would inevitably diverge from each other.

Have a look at [helm](https://helm.sh/). You can use it to bundle templates for k8s and fill them at deploy time. — fishi0x01, Dec 05 '17 at 15:06

score 10 · Accepted Answer · edited Jun 20 '20 at 09:12

Helm

Helm is becoming the standard for templatizing kubernetes deployments. A helm chart is a directory consisting of yaml files with golang variable placeholders

---
kind: Deployment
metadata:
  name: foo
spec:
  replicas: {{ .Values.replicaCount }}

You define the default value of a 'value' in the 'values.yaml' file

replicaCount: 1

You can optionally overwrite the value using the --set command line

helm install foo --set replicaCount=42

Helm can also point to an external answer file

helm install foo -f ./dev.yaml
helm install foo -f ./prod.yaml

dev.yaml

---
replicaCount: 1

prod.yaml

---
replicaCount: 42

Another advantage of Helm over simpler solutions like envbsubst is that Helm supports plugins. One powerful plugin is the helm-secrets plugin that lets you encrypt sensitive data using pgp keys. https://github.com/futuresimple/helm-secrets

If using helm + helm-secrets your setup may look like the following where your code is in one repo and your data is in another.

git repo with helm charts

stable
  |__mysql
     |__Values.yaml
     |__Charts
  |__apache
     |__Values.yaml
     |__Charts
incubator
  |__mysql
     |__Values.yaml
     |__Charts
  |__apache
     |__Values.yaml
     |__Charts

Then in another git repo that contains the environment specific data

values
|__ mysql
    |__dev
       |__values.yaml
       |__secrets.yaml
    |__prod
       |__values.yaml
       |__secrets.yaml

You then have a wrapper script that references the values and the secrets files

helm secrets upgrade foo --install -f ./values/foo/$environment/values.yaml -f ./values/foo/$environment/secrets.yaml

envsubst

As mentioned in other answers, envsubst is a very powerful yet simple way to make your own templates. An example from kiminehart

apiVersion: extensions/v1beta1
kind: Deployment
# ...
    architecture: ${GOOS}

GOOS=amd64 envsubst < mytemplate.tmpl > mydeployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
# ...
    architecture: amd64

Kubectl

There is a feature request to allow kubectl to do some of the same features of helm and allow for variable substitution. There is a background document that strongly suggest that the feature will never be added, and instead is up to external tools like Helm and envsubst to manage templating.

(edit)

Kustomize

Kustomize is a new project developed by google that is very similar to helm. Basically you have 2 folders base and overlays. You then run kustomize build someapp/overlays/production and it will generate the yaml for that environment.

   someapp/
   ├── base/
   │   ├── kustomization.yaml
   │   ├── deployment.yaml
   │   ├── configMap.yaml
   │   └── service.yaml
   └── overlays/
      ├── production/
      │   └── kustomization.yaml
      │   ├── replica_count.yaml
      └── staging/
          ├── kustomization.yaml
          └── cpu_count.yaml

It is simpler and has less overhead than helm, but does not have plugins for managing secrets. You could combine kustomize with sops or envsubst to manage secrets.

https://kubernetes.io/blog/2018/05/29/introducing-kustomize-template-free-configuration-customization-for-kubernetes/

score 3 · Answer 2 · answered Dec 05 '17 at 14:51

I'm hoping someone will give me a better answer, but in the meantime, you can feed your configuration through envsubst (see gettext and this for mac).

Example config, text.yaml:

apiVersion: apps/v1beta2
kind: Deployment
metadata:
  name: test
spec:
  replicas: ${NUM_REPLICAS}
  ...

Then run:

$ NUM_REPLICAS=2 envsubst < test.yaml | kubectl apply -f -
deployment "test" configured

The final dash is required. This doesn't solve the problem with volumes of course, but it helps a little. You could write a script/makefile to automate this for environment.

Adjusting Kubernetes configurations depending on environment

2 Answers2

Helm

envsubst

Kubectl

Kustomize