K8s jobs and pods differences as uses of host+subdomain

Question

I have K8s used by Helm 3.

1. I need to access a k8s job while running in yaml file (created by helm).

The kubectl version:

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6", GitCommit:"d921bc6d1810da51177fbd0ed61dc811c5228097", GitTreeState:"clean", BuildDate:"2021-10-27T17:50:34Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.6", GitCommit:"d921bc6d1810da51177fbd0ed61dc811c5228097", GitTreeState:"clean", BuildDate:"2021-10-27T17:44:26Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"linux/amd64"}

Helm version:

version.BuildInfo{Version:"v3.3.4", GitCommit:"a61ce5633af99708171414353ed49547cf05013d", GitTreeState:"clean", GoVersion:"go1.14.9"}

As the following link: DNS concept

It works fine for Pod, but not for job.

As explained, for putting hostname and subdomain in Pod's YAML file, and add service that holds the domain...

2. Need to check the state if running.

for pod, it is ready state.

kubectl wait pod/pod-name --for=condition=ready ...

For job there is no ready state (while pod behind is running).

How can I check the state of pod behind the job (job is running) and how can I use host + subdomain for jobs?

My code ... (I removed some security tags, but the same. Important - It may be complicated.

I create a listener - running when listen, with job that need to do some curl command, and this can be achieved whether it has access to that pod behind the job):

Listener (the pod is the last job):

What I added is hostname and subdomain (which work for Pod, and not for Job). If it ever was on Pod - no problem.

I also realized that the name of the Pod (created by the job) has a hash automatic extension.

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "my-project.fullname" . }}-listener
  namespace: {{ .Release.Namespace }}
  labels:
    name: {{ include "my-project.fullname" . }}-listener
    app: {{ include "my-project.fullname" . }}-listener
    component: {{ .Chart.Name }}
    subcomponent: {{ .Chart.Name }}-listener
  annotations:
    "prometheus.io/scrape": {{ .Values.prometheus.scrape | quote }}
    "prometheus.io/path": {{ .Values.prometheus.path }}
    "prometheus.io/port": {{ .Values.ports.api.container | quote }}
spec:
  template: #PodTemplateSpec (Core/V1)
    spec: #PodSpec (core/v1)
      hostname: {{ include "my-project.fullname" . }}-listener
      subdomain: {{ include "my-project.fullname" . }}-listener-dmn
      initContainers:
        # twice - can add in helers.tpl
        - name: wait-mysql-exist-pod
          image: {{ .Values.global.registry }}/{{ .Values.global.k8s.image }}:{{ .Values.global.k8s.tag | default "latest" }}
          imagePullPolicy: IfNotPresent
          env:
            - name: MYSQL_POD_NAME
              value: {{ .Release.Name }}-mysql
            - name: COMPONENT_NAME
              value: {{ .Values.global.mysql.database.name }}
          command:
            - /bin/sh
          args:
            - -c
            - |-
              while [ "$(kubectl get pod $MYSQL_POD_NAME 2>/dev/null | grep $MYSQL_POD_NAME | awk '{print $1;}')" \!= "$MYSQL_POD_NAME" ];do
                echo 'Waiting for mysql pod to be existed...';
                sleep 5;
              done
        - name: wait-mysql-ready
          image: {{ .Values.global.registry }}/{{ .Values.global.k8s.image }}:{{ .Values.global.k8s.tag | default "latest" }}
          imagePullPolicy: IfNotPresent
          env:
            - name: MYSQL_POD_NAME
              value: {{ .Release.Name }}-mysql
          command:
            - kubectl
          args:
            - wait
            - pod/$(MYSQL_POD_NAME)
            - --for=condition=ready
            - --timeout=120s
        - name: wait-mysql-has-db
          image: {{ .Values.global.registry }}/{{ .Values.global.k8s.image }}:{{ .Values.global.k8s.tag | default "latest" }}
          imagePullPolicy: IfNotPresent
          env:
            {{- include "k8s.db.env" . | nindent 12 }}
            - name: MYSQL_POD_NAME
              value: {{ .Release.Name }}-mysql
          command:
            - /bin/sh
          args:
            - -c
            - |-
             while [ "$(kubectl exec $MYSQL_POD_NAME -- mysql -uroot -p$MYSQL_ROOT_PASSWORD -e 'show databases' 2>/dev/null | grep $MYSQL_DATABASE | awk '{print $1;}')" \!= "$MYSQL_DATABASE" ]; do
                echo 'Waiting for mysql database up...';
                sleep 5;
             done
      containers:
        - name: {{ include "my-project.fullname" . }}-listener
          image:  {{ .Values.global.registry }}/{{ .Values.image.repository }}:{{ .Values.image.tag | default "latest" }}
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          env:
          {{- include "k8s.db.env" . | nindent 12 }}
            - name: SCHEDULER_DB
              value: $(CONNECTION_STRING)
          command: {{- toYaml .Values.image.entrypoint | nindent 12 }}
          args: # some args ...
          ports:
            - name: api
              containerPort: 8081
          resources:
            limits:
              cpu: 1
              memory: 1024Mi
            requests:
              cpu: 100m
              memory: 50Mi
          readinessProbe:
            httpGet:
              path: /api/scheduler/healthcheck
              port: api
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 5
            timeoutSeconds: 1
          livenessProbe:
            tcpSocket:
              port: api
            initialDelaySeconds: 120
            periodSeconds: 10
            timeoutSeconds: 5
          volumeMounts:
            - name: {{ include "my-project.fullname" . }}-volume
              mountPath: /etc/test/scheduler.yaml
              subPath: scheduler.yaml
              readOnly: true
      volumes:
      - name: {{ include "my-project.fullname" . }}-volume
        configMap:
          name: {{ include "my-project.fullname" . }}-config
      restartPolicy: Never

The service (for the subdomain):

apiVersion: v1
kind: Service
metadata:
  name: {{ include "my-project.fullname" . }}-listener-dmn
spec:
  selector:
    name: {{ include "my-project.fullname" . }}-listener
  ports:
    - name: api
      port: 8081
      targetPort: 8081
  type: ClusterIP

Roles + RoleBinding (to enable access for curl command):

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: {{ include "my-project.fullname" . }}-role
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods"]
  verbs: ["get", "watch", "list", "update"]
- apiGroups: [""] # "" indicates the core API group
  resources: ["pods/exec"]
  verbs: ["create", "delete", "deletecollection", "get", "list", "patch", "update", "watch"]
- apiGroups: ["", "app", "batch"] # "" indicates the core API group
  resources: ["jobs"]
  verbs: ["get", "watch", "list"]

Role-Binding:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: {{ include "go-scheduler.fullname" . }}-rolebinding
subjects:
- kind: ServiceAccount
  name: default
roleRef:
  kind: Role
  name: {{ include "go-scheduler.fullname" . }}-role
  apiGroup: rbac.authorization.k8s.io

And finally a tester that doing a curl command:

(For check I put tail -f), and enter the pod.

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "my-project.fullname" . }}-test
  namespace: {{ .Release.Namespace }}
  labels:
    name: {{ include "my-project.fullname" . }}-test
    app: {{ include "my-project.fullname" . }}-test 
  annotations:
    "prometheus.io/scrape": {{ .Values.prometheus.scrape | quote }}
    "prometheus.io/path": {{ .Values.prometheus.path }}
    "prometheus.io/port": {{ .Values.ports.api.container | quote }}
spec:
  template: #PodTemplateSpec (Core/V1)
    spec: #PodSpec (core/v1)
      initContainers:
        # twice - can add in helers.tpl
        #
        - name: wait-sched-listener-exists
          image: {{ .Values.global.registry }}/{{ .Values.global.k8s.image }}:{{ .Values.global.k8s.tag | default "latest" }}
          imagePullPolicy: IfNotPresent
          env:
            - name: POD_NAME
              value: {{ include "my-project.fullname" . }}-listener
          command:
            - /bin/sh
          args:
            - -c
            - |-
              while [ "$(kubectl get job $POD_NAME 2>/dev/null | grep $POD_NAME | awk '{print $1;}')" \!= "$POD_NAME" ];do
                echo 'Waiting for scheduler pod to exist ...';
                sleep 5;
              done
        - name: wait-listener-running
          image: {{ .Values.global.registry }}/{{ .Values.global.k8s.image }}:{{ .Values.global.k8s.tag | default "latest" }}
          imagePullPolicy: IfNotPresent
          env:
            - name: POD_NAME
              value: {{ include "my-project.fullname" . }}-listener
          command:
            - /bin/sh
          args:
            - -c
            - |-
              while [ "$(kubectl get pods 2>/dev/null | grep $POD_NAME | awk '{print $3;}')" \!= "Running" ];do
                echo 'Waiting for scheduler pod to run ...';
                sleep 5;
              done
      containers:
        - name: {{ include "my-project.fullname" . }}-test
          image:  {{ .Values.global.registry }}/{{ .Values.global.k8s.image }}:{{ .Values.global.k8s.tag | default "latest" }}
          imagePullPolicy: {{ .Values.image.pullPolicy }}
          command:
            - /bin/sh
          args:
            - -c
            - "tail -f"
     # instead of above can be curl: "curl -H 'Accept: application/json' -X get my-project-listener.my-project-listener-dmn:8081/api/scheduler/jobs"

      restartPolicy: Never

I enter the test pod

kubectl exec -it my-tester-<hash> -- /bin/sh

... and run the command:

ping my-project-listener.my-project-listener-dmn

Got:

ping: bad address 'my-project-listener.my-project-listener-dmn'

When doing that for pod:

PING pod-hostname.pod-subdomain (): ... data bytes

Answer 1

There's a lot here, but I think you should be able to resolve all of this with a couple of small changes.

In summary, I'd suggest changing:

apiVersion: apps/v1
kind: Deployment     # <-- not a Job
metadata: &original-job-metadata-from-the-question
spec:
  template:
    metadata:
      labels:   # vvv matching the Service selector
        name: {{ include "my-project.fullname" . }}-listener
    spec:
      # delete all of the initContainers:
      containers: &original-container-list-from-the-question
      volumes: &original-volume-list-from-the-question
      # delete restartPolicy: (default value Always)

Delete the Role and RoleBinding objects; connect to the Service http://my-project-listener-dmn:8081 and not an individual Pod; and you can kubectl wait --for=condition=available on the Deployment.

Connect to Services, not individual Pods (or Jobs or Deployments). The Service is named {{ include "my-project.fullname" . }}-listener-dmn and that is the host name you should connect to. The Service acts as a very lightweight in-cluster load balancer, and will forward requests on to one of the pods identified by its selector.

So in this example you'd connect to the Service's name and port, http://my-project-listener-dmn:8081. Your application doesn't answer the very-low-level ICMP protocol and I'd avoid ping(1) in favor of a more useful diagnostic. Also consider setting the Service's port to the default HTTP port 80; it doesn't necessarily need to match the Pod's port.

The Service selector needs to match the Pod labels (and not the Job's or Deployment's labels). A Service attaches to Pods; a Job or a Deployment has a template to create Pods; and it's those labels that need to match up. You need to add labels to the Pod template:

spec:
  template:
    metadata:
      labels:
        name: {{ include "my-project.fullname" . }}-listener

Or, in a Helm chart where you have a helper to generate these labels,

      labels: {{- include "my-project.labels" | nindent 8 }}

The thing to check here is kubectl describe service my-project-listener-dmn. There should be a line at the bottom that says Endpoints: with some IP addresses (technically some individual Pod IP addresses, but you don't usually need to know that). If it says Endpoints: <none> that's usually a sign that the labels don't match up.

You probably want some level of automatic restarts. A Pod can fail for lots of reasons, including code bugs and network hiccups. If you set restartPolicy: Never then you'll have a Failed Pod, and requests to the Service will fail until you take manual intervention of some sort. I'd suggest setting this to at least restartPolicy: OnFailure, or (for a Deployment) leaving it at its default value of Always. (There is more discussion on Job restart policies in the Kubernetes documentation.)

You probably want a Deployment here. A Job is meant for a case where you do some set of batch processing and then the job completes; that's part of why kubectl wait doesn't have the lifecycle option you're looking for.
I'm guessing you want a Deployment instead. With what you've shown here I don't think you need to make any changes at all besides

apiVersion: apps/v1
kind: Deployment

Everything so far about Services and DNS and labels still applies.

You can kubectl wait for a Deployment to be available. Since a Job is expected to run to completion and exit, that's the state kubectl wait allows. A Deployment is "available" if there is at least a minimum number of managed Pods running that pass their health checks, which I think is the state you're after.

kubectl wait --for=condition=available deployment/my-project-listener

There are simpler ways to check for database liveness. A huge fraction of what you show here is an involved sequence with special permissions to see if the database is running before the pod starts up.

What happens if the database fails while the pod is running? One common thing that will happen is you'll get a cascading sequence of exceptions and your pod will crash. Then with restartPolicy: Always Kubernetes will try to restart it; but if the database still isn't available, it will crash again; and you'll get to a CrashLoopBackOff state. If the database does become available again then eventually Kubernetes will try to restart the Pod and it will succeed.

This same logic can apply at startup time. If the Pod tries to start up, and the database isn't ready yet, and it crashes, Kubernetes will by default restart it, adding some delays after the first couple of attempts. If the database starts up within 30 seconds or so then the application will be up within a minute or so. The restart count will be greater than 0, but kubectl logs --previous will hopefully have a clear exception.

This will let you delete about half of what you show here. Delete all of the initContainers: block; then, since you're not doing any Kubernetes API operations, delete the Role and RoleBinding objects too.

If you really do want to force the Pod to wait for the database and treat startup as a special case, I'd suggest a simpler shell script using the mysql client tool, or even the wait-for script that makes basic TCP calls (the mechanism described in Docker Compose wait for container X before starting Y). This still lets you avoid all of the Kubernetes RBAC setup.