When a pod can't be scheduled, what does the 3 in "Insufficient cpu (3)" refer to?

Question

When I create a Pod that cannot be scheduled because there are no nodes with sufficient CPU to meet the Pod's CPU request, the events output from kubectl describe pod/... contain a message like No nodes are available that match all of the following predicates:: Insufficient cpu (3).

What does the (3) in Insufficient cpu (3) mean?

For example, if I try to create a pod that requests 24 CPU when all of my nodes only have 4 CPUs:

$ kubectl describe pod/large-cpu-request
Name:           large-cpu-request
Namespace:      default
Node:           /
Labels:         <none>
Annotations:    <none>
Status:         Pending
IP:
Controllers:    <none>
Containers:
  cpuhog:
    ...
    Requests:
      cpu:              24
...
Events:
  FirstSeen     LastSeen        Count   From                    SubObjectPath   Type            Reason                  Message
  ---------     --------        -----   ----                    -------------   --------        ------                  -------
  23m           30s             84      default-scheduler                       Warning         FailedScheduling        No nodes are available that match all of the following predicates:: Insufficient cpu (3).

At other times I have seen event messages like No nodes are available that match all of the following predicates:: Insufficient cpu (2), PodToleratesNodeTaints (1) when a pod's resource requests were too high, so the 3 does not seem like a constant number - nor does it seem related to my 24 CPU request either.

Answer 1

It means that your Pod doesn't fit on 3 nodes because of Insufficient CPU and 1 node because of taints (likely the master).

Answer 2

A pod can't be scheduled when it requests more CPU than you have in your cluster. For example, if you have 8 Kubernetes CPUs (see this page to calculate how many Kubernetes CPUs you have) in total and if your existing pods have already consumed that much CPU, then you can't schedule more pods unless some of your existing pods are killed by the time you request to schedule a new pod. Here is a simple equation that can be followed in Horizontal Pod Autoscaler (HPA):

RESOURCE REQUEST CPU * HPA MAX PODS <= Total Kubernetes CPU

You can always tune up these numbers. In my case, I adjusted my manifest file for the RESOURCE REQUEST CPU. So it can be 200m or 1000m (= 1 Kubernetes CPU).