Application pod status remain in pending

Question

I have deployment an application, But pod always in pending state.

$ kubectl get nodes
NAME          STATUS   ROLES           AGE   VERSION
server1       Ready    control-plane   8d    v1.24.9
server2       Ready    worker1         8d    v1.24.9
server3       Ready    worker2         8d    v1.24.9
server4       Ready    worker3         8d    v1.24.9

$ kubectl get all -n jenkins
NAME                          READY   STATUS    RESTARTS   AGE
pod/jenkins-6dc9f97c7-ttp64   0/1     Pending   0          7m42s

$ kubectl describe pods jenkins-6dc9f97c7-ttp64 -n jenkins

Events:
Type     Reason            Age    From               Message
----     ------            ----   ----               -------
Warning  FailedScheduling  5m42s  default-scheduler  0/4 nodes are available: 3 node(s) had volume node affinity conflict, 4 node(s) didn't match Pod's node affinity/selector. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling.

The event history confirms a FailedScheduling error is the reason.

my deployment.yml has forced to assign the pod into master node.

 spec:
      nodeSelector:
        node-role.kubernetes.io/master: ""
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists

Since from Kubernetes version 1.20+ onwards node-role.kubernetes.io/master is deprecated in favor of node-role.kubernetes.io/control-plane i have updated like below. However still pod showing as pending.

    spec:
      nodeSelector:
        node-role.kubernetes.io/control-plane: ""
      tolerations:
      - key: node-role.kubernetes.io/control-plane

PersistentVolume.yml side i have below content.

...
.....
..........
  local:
    path: /ksdata/apps/nodejs/
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - server1

Label details:-

$ kubectl get nodes --show-labels
NAME      STATUS   ROLES           AGE   VERSION   LABELS
server1   Ready    control-plane   9d    v1.24.9   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=server1,kubernetes.io/os=linux,node-role.kubernetes.io/control-plane=,node.kubernetes.io/exclude-from-external-load-balancers=
server2   Ready    worker1         9d    v1.24.9   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=server2,kubernetes.io/os=linux,node-role.kubernetes.io/worker1=worker
server3   Ready    worker2         9d    v1.24.9   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=server3,kubernetes.io/os=linux,node-role.kubernetes.io/worker2=worker
server4   Ready    worker3         9d    v1.24.9   beta.kubernetes.io/arch=amd64,beta.kubernetes.io/os=linux,kubernetes.io/arch=amd64,kubernetes.io/hostname=server4,kubernetes.io/os=linux,node-role.kubernetes.io/worker3=worker

$ kubectl describe node | egrep -i taint
Taints:             key=value:NoSchedule
Taints:             <none>
Taints:             <none>
Taints:             <none>

Harsh Manvar · Accepted Answer · 2023-01-19T09:41:46.523

1

You have 4 nodes in the cluster, so generally one is master node, and on top of that app pods not schedule so 3 nodes are left.

While with the worker node, your deployment has of node affinity set so due to that it can't schedule a pod on that node and it's stuck in a pending state.

Check the PVC mostly it won't be able to get created

Update

Remove taint form master or control-plane

kubectl taint node server1 key=value:NoSchedule-

Toleration to set on master

  spec:
    nodeSelector:
        kubernetes.io/hostname: "server1"

#if have tain and not removed try toleration else fine with nodeselector

tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule

edited Jan 19 '23 at 09:41

answered Jan 18 '23 at 11:01

Harsh Manvar

27,020
6
48
102

your jenkin pod have any pvc or volume disk ? that might be having some node affinity config in yaml remove it try to satisfy that rule and it will start. – Harsh Manvar Jan 18 '23 at 11:04
if possible dont add the nodeselector at all, by default it wills schedule on any available worker node unless you have requirement to run that pod on specific node that time we use the nodeselector. else it fine to use the without that also you have kept is empty as value instead better not to add. it will choose suitable node – Harsh Manvar Jan 19 '23 at 05:18
Actually for project specific purpose. pod needs to schedule on `control-plane` node. – user4948798 Jan 19 '23 at 05:20
if you are adding tolertation why adding it nodeselector ? read official doc how to use the toleration and taints :https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/ – Harsh Manvar Jan 19 '23 at 05:24
you might have to check the taint of master if any set and accordingly have to add toleration to POD, ref this https://stackoverflow.com/questions/43147941/allow-scheduling-of-pods-on-kubernetes-master – Harsh Manvar Jan 19 '23 at 05:26
`tolertation` used to allow the scheduler to schedule pods with matching `control-plane` and `nodeselector` field added to match the specified volume. – user4948798 Jan 19 '23 at 05:36
did you check the doc : https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes/ you are passing label value empty although have set the label not sure it will work or not as empty value of label – Harsh Manvar Jan 19 '23 at 05:37
first, check your node has which taint set master or control plane? also share the label master has set now. – Harsh Manvar Jan 19 '23 at 06:56
from pod defination either use the node selector or use toleration not both, if using node selector use label which is set on master if using toleration use the proper taint which is set. – Harsh Manvar Jan 19 '23 at 06:58
`kubectl get nodes --show-labels` output added in question section. – user4948798 Jan 19 '23 at 07:37
share the tain output also, if possinle else update answer check node selector and toleration and delete pod once and check if getting schedule or stuck in pending – Harsh Manvar Jan 19 '23 at 07:40
`kubectl describe node | egrep -i taint` output added in question. – user4948798 Jan 19 '23 at 07:44
you can use my answer suggestion and give it try toleration and node selector in deployment deifiniation adding both and give it try – Harsh Manvar Jan 19 '23 at 08:06
no success. still same problem getting `0/4 nodes are available: 1 node(s) had untolerated taint {key: value}, 3 node(s) didn't match Pod's node affinity/selector, 3 node(s) had volume node affinity conflict. preemption: 0/4 nodes are available: 4 Preemption is not helpful for scheduling` – user4948798 Jan 19 '23 at 08:52
remove the taint from node `kubectl taint node server1 key=value:NoSchedule-` – Harsh Manvar Jan 19 '23 at 09:10

Application pod status remain in pending

1 Answers1