0

I created a k8 Job to use for schema migration in a rails app. Below is the Job yaml excluding the env vars:

---
- name: Deploy Migration Worker
  k8s:
    state: present
    force: 'yes'
    definition:
      apiVersion: batch/v1
      kind: Job
      metadata:
        name: schema-migration
        namespace: "{{ k8s_namespace }}"
      spec:
        ttlSecondsAfterFinished: 60
        template:
          spec:
            containers:
              - command:
                  - /bin/sh
                args:
                  - '-c'
                  - '{{ MIGRATIONS }}'
                name: schema-migration-container
                resources:
                  limits:
                    cpu: "{{ API_LIMIT_CPU }}"
                    memory: "{{ API_LIMIT_MEM }}"
                  requests:
                    cpu: "{{ API_REQUEST_CPU }}"
                    memory: "{{ API_REQUEST_MEM }}"
                image: "redacted"
                imagePullPolicy: IfNotPresent
            restartPolicy: Never
            imagePullSecrets:
              - name: docker-pull-secret
            dnsPolicy: ClusterFirst
            imagePullSecrets:
              - name: docker-pull-secret
            schedulerName: default-scheduler
            securityContext: {}
            terminationGracePeriodSeconds: 30

Below is the pod status:

NAME                            READY   STATUS     RESTARTS   AGE
schema-migration-mnvvw          1/2     NotReady   0          137m

Below is the job status:

NAME               COMPLETIONS   DURATION   AGE
schema-migration   0/1           133m       133m

Below is the pod description:

 Name:         schema-migration-mnvvw
Namespace:    dev1
Priority:     0
Node:         redacted
Start Time:   Wed, 01 Feb 2023 15:16:35 -0400
Labels:       controller-uid=redacted
              job-name=schema-migration
              security.istio.io/tlsMode=istio
              service.istio.io/canonical-name=schema-migration
              service.istio.io/canonical-revision=latest
Annotations:  cluster-autoscaler.kubernetes.io/safe-to-evict: true
              kubectl.kubernetes.io/default-container: main
              kubectl.kubernetes.io/default-logs-container: main
              kubernetes.io/psp: eks.privileged
              prometheus.io/path: /stats/prometheus
              prometheus.io/port: 15020
              prometheus.io/scrape: true
              sidecar.istio.io/status:
                {"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-data","istio-podinfo","istio-token","istiod-...
Status:       Running
IP:           10.131.217.49
IPs:
  IP:           10.131.217.49
Controlled By:  Job/schema-migration
Init Containers:
  istio-init:
    Container ID:  redacted
    Image:         docker.io/istio/proxyv2:1.11.3
    Image ID:      docker-pullable://istio/proxyv2@sha256:28513eb3706315b26610a53e0d66b29b09a334e3164393b9a0591f34fe47a6fd
    Port:          <none>
    Host Port:     <none>
    Args:
      istio-iptables
      -p
      15001
      -z
      15006
      -u
      1337
      -m
      REDIRECT
      -i
      *
      -x
      172.20.0.1/32
      -b
      *
      -d
      15090,15021,15020
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 01 Feb 2023 15:16:36 -0400
      Finished:     Wed, 01 Feb 2023 15:16:36 -0400
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:     100m
      memory:  128Mi
    Environment:
      DNS_AGENT:  
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9vcj4 (ro)
Containers:
  main:
    Container ID:  docker://a37824a15a748e7124a455a878f57b2ae22e08f8cddd0a2b1938b0414228b320
    Image:         redacted
    Image ID:      redacted
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/sh
    Args:
      -c
      bundle exec rake db:migrate:status && bundle exec rake db:migrate && bundle exec rake seed:policy_types && bundle exec rake seed:permission_types && bundle exec rake seed:roles && bundle exec rake seed:default_role_permissions && bundle exec rake seed:partner_policy_defaults && bundle exec rake after_party:run
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Wed, 01 Feb 2023 15:16:50 -0400
      Finished:     Wed, 01 Feb 2023 15:18:13 -0400
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     1
      memory:  4G
    Requests:
      cpu:      250m
      memory:   2G
    Readiness:  http-get http://:15020/app-health/main/readyz delay=30s timeout=1s period=5s #success=1 #failure=12
    Mounts:
      /etc/istio/pod from istio-podinfo (rw)
      /etc/istio/proxy from istio-envoy (rw)
      /var/lib/istio/data from istio-data (rw)
      /var/run/secrets/istio from istiod-ca-cert (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9vcj4 (ro)
      /var/run/secrets/tokens from istio-token (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  istio-envoy:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  istio-data:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  istio-podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
      metadata.annotations -> annotations
  istio-token:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  43200
  istiod-ca-cert:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      istio-ca-root-cert
    Optional:  false
  kube-api-access-9vcj4:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

I'm fairly new to kubernetes however I've tried adding/removing a readinessProbe attribute as well as increasing cpu/memory resources among other things with no luck. There seems to be only a handful of similar issues (pod status showing NotReady as opposed to node status) but none that seems related to my problem. One thing of note perhaps, is that this is created via Ansible playbook though I don't believe this comes into play as far as this issue goes.

I'm hoping to get the job to a completed state essentially. Thanks in advance.

  • 2
    It looks like you're using Istio. There's a fairly long-standing problem that the Istio sidecar won't exit on its own when the main Job task completes, which keeps the Pod alive; see for example [Terminate istio sidecar istio-proxy for a kubernetes job / cronjob](https://stackoverflow.com/questions/54921054/terminate-istio-sidecar-istio-proxy-for-a-kubernetes-job-cronjob). [How to disable istio on K8S job](https://stackoverflow.com/questions/59235887/how-to-disable-istio-on-k8s-job) is another possible alternative. – David Maze Feb 02 '23 at 02:07

0 Answers0