I created a k8 Job to use for schema migration in a rails app. Below is the Job yaml excluding the env vars:
---
- name: Deploy Migration Worker
k8s:
state: present
force: 'yes'
definition:
apiVersion: batch/v1
kind: Job
metadata:
name: schema-migration
namespace: "{{ k8s_namespace }}"
spec:
ttlSecondsAfterFinished: 60
template:
spec:
containers:
- command:
- /bin/sh
args:
- '-c'
- '{{ MIGRATIONS }}'
name: schema-migration-container
resources:
limits:
cpu: "{{ API_LIMIT_CPU }}"
memory: "{{ API_LIMIT_MEM }}"
requests:
cpu: "{{ API_REQUEST_CPU }}"
memory: "{{ API_REQUEST_MEM }}"
image: "redacted"
imagePullPolicy: IfNotPresent
restartPolicy: Never
imagePullSecrets:
- name: docker-pull-secret
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: docker-pull-secret
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
Below is the pod status:
NAME READY STATUS RESTARTS AGE
schema-migration-mnvvw 1/2 NotReady 0 137m
Below is the job status:
NAME COMPLETIONS DURATION AGE
schema-migration 0/1 133m 133m
Below is the pod description:
Name: schema-migration-mnvvw
Namespace: dev1
Priority: 0
Node: redacted
Start Time: Wed, 01 Feb 2023 15:16:35 -0400
Labels: controller-uid=redacted
job-name=schema-migration
security.istio.io/tlsMode=istio
service.istio.io/canonical-name=schema-migration
service.istio.io/canonical-revision=latest
Annotations: cluster-autoscaler.kubernetes.io/safe-to-evict: true
kubectl.kubernetes.io/default-container: main
kubectl.kubernetes.io/default-logs-container: main
kubernetes.io/psp: eks.privileged
prometheus.io/path: /stats/prometheus
prometheus.io/port: 15020
prometheus.io/scrape: true
sidecar.istio.io/status:
{"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["istio-envoy","istio-data","istio-podinfo","istio-token","istiod-...
Status: Running
IP: 10.131.217.49
IPs:
IP: 10.131.217.49
Controlled By: Job/schema-migration
Init Containers:
istio-init:
Container ID: redacted
Image: docker.io/istio/proxyv2:1.11.3
Image ID: docker-pullable://istio/proxyv2@sha256:28513eb3706315b26610a53e0d66b29b09a334e3164393b9a0591f34fe47a6fd
Port: <none>
Host Port: <none>
Args:
istio-iptables
-p
15001
-z
15006
-u
1337
-m
REDIRECT
-i
*
-x
172.20.0.1/32
-b
*
-d
15090,15021,15020
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 01 Feb 2023 15:16:36 -0400
Finished: Wed, 01 Feb 2023 15:16:36 -0400
Ready: True
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 100m
memory: 128Mi
Environment:
DNS_AGENT:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9vcj4 (ro)
Containers:
main:
Container ID: docker://a37824a15a748e7124a455a878f57b2ae22e08f8cddd0a2b1938b0414228b320
Image: redacted
Image ID: redacted
Port: <none>
Host Port: <none>
Command:
/bin/sh
Args:
-c
bundle exec rake db:migrate:status && bundle exec rake db:migrate && bundle exec rake seed:policy_types && bundle exec rake seed:permission_types && bundle exec rake seed:roles && bundle exec rake seed:default_role_permissions && bundle exec rake seed:partner_policy_defaults && bundle exec rake after_party:run
State: Terminated
Reason: Completed
Exit Code: 0
Started: Wed, 01 Feb 2023 15:16:50 -0400
Finished: Wed, 01 Feb 2023 15:18:13 -0400
Ready: False
Restart Count: 0
Limits:
cpu: 1
memory: 4G
Requests:
cpu: 250m
memory: 2G
Readiness: http-get http://:15020/app-health/main/readyz delay=30s timeout=1s period=5s #success=1 #failure=12
Mounts:
/etc/istio/pod from istio-podinfo (rw)
/etc/istio/proxy from istio-envoy (rw)
/var/lib/istio/data from istio-data (rw)
/var/run/secrets/istio from istiod-ca-cert (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9vcj4 (ro)
/var/run/secrets/tokens from istio-token (rw)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
istio-envoy:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium: Memory
SizeLimit: <unset>
istio-data:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
istio-podinfo:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels -> labels
metadata.annotations -> annotations
istio-token:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 43200
istiod-ca-cert:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: istio-ca-root-cert
Optional: false
kube-api-access-9vcj4:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events: <none>
I'm fairly new to kubernetes however I've tried adding/removing a readinessProbe
attribute as well as increasing cpu/memory resources among other things with no luck. There seems to be only a handful of similar issues (pod status showing NotReady as opposed to node status) but none that seems related to my problem. One thing of note perhaps, is that this is created via Ansible playbook though I don't believe this comes into play as far as this issue goes.
I'm hoping to get the job to a completed state essentially. Thanks in advance.