introducing delay for running helm test suite over installed release

Question

I'm using helm charts to deploy several REST services with into k8s with spring boot inside deployed container.

However, to be able to do final stage testing I need to introduce some sort of smart liveness probe - i.e. that the target application is actually running properly inside given container.

This can be easily justified by successful return code of simple curl command, however, here's the trick - command needs to be executed with some delay after particular release deployment to give application time to bootstrap.

Here's what I've figured for a test suite:

apiVersion: v1
kind: Pod
metadata:
  name: "{{ include "chart.fullname" . }}-test"
  labels:
  {{- include "chart.fullname" . | nindent 4 }}
  annotations:
    "helm.sh/hook": test-success
spec:
  containers:
    - name: test-curl
      image: curl
      command: ['curl']
      args: [' -X POST -i -H "Accept: application/json" -H "Content-Type:application/json" -d ''{"foo":["bar"]}'' {{ include "chart.fullname" . }}:{{ .Values.service.port }}']
  restartPolicy: Never

The key problem is that this test will be executed when service is not really started yet and thus fail anyway.

Is there some mechanism or workaround to introduce delay of execution for this test?

Setting some sleep step in separate test container comes to mind but i'm not sure if this will work properly for this case

You should introduce retry logic in the test command. Instead of directly running the curl command, run a sh/bash shell and add a retry logic there. This thread might help. https://unix.stackexchange.com/a/82602 — Emruz Hossain, Jul 08 '21 at 08:10
@EmruzHossain so the release mechanism will prevent new release rolling until test will complete? — im_infamous, Jul 08 '21 at 08:27
Do you have a [readiness probe](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) configured on your application? That should cause `helm install --wait` to wait until the application is alive, and then you can run `helm test`. — David Maze, Jul 08 '21 at 10:02

score 2 · Answer 1 · answered Jul 08 '21 at 08:36

2

curl support retry the request via --retry flag. You can pass this flag in as command args. You can check more configurable options for retry from this thread: https://stackoverflow.com/a/42873372/7695859

answered Jul 08 '21 at 08:36

Emruz Hossain

4,764
18
26

score 2 · Accepted Answer · answered Jul 15 '21 at 16:10

thanks to @Emruz Hossain, I've figured the solution out:

apiVersion: batch/v1
kind: Job
metadata:
  name: "{{ .Release.Name }}-test"
  labels:
    app: {{ .Release.Name }}
    release: {{ .Release.Name }}
  annotations:
    "helm.sh/hook": test-success
spec:
  ttlSecondsAfterFinished: 0
  template:
    spec:
      containers:
        - name: test-curl
          image: target-image:1.2.3
          imagePullPolicy: "IfNotPresent"
          command:
            - /bin/bash
            - -ec
            - |
              curl --connect-timeout 5 --max-time 10 --retry 5 --retry-delay 5 --retry-max-time 30 --retry-all-errors http://{{ .Release.Name }}:{{ .Values.service.port }}/v1/rest -X POST -H "content-type: application/json" -d "{\"foo\":[\"bar\"]}"
      restartPolicy: Never

requires k8s api server 1.20+ (due to this) and curl 7.71+ (due to this)

introducing delay for running helm test suite over installed release

2 Answers2