GKE Insufficient CPU for small Node.js app pods

Question

So on GKE I have a Node.js app which for each pod uses about: CPU(cores): 5m, MEMORY: 100Mi

However I am only able to deploy 1 pod of it per node. I am using the GKE n1-standard-1 cluster which has 1 vCPU, 3.75 GB per node.

So in order to get 2 pods of app up total = CPU(cores): 10m, MEMORY: 200Mi, it requires another entire +1 node = 2 nodes = 2 vCPU, 7.5 GB to make it work. If I try to deploy those 2 pods on the same single node, I get insufficient CPU error.

I have a feeling I should actually be able to run a handful of pod replicas (like 3 replicas and more) on 1 node of f1-micro (1 vCPU, 0.6 GB) or f1-small (1 vCPU, 1.7 GB), and that I am way overprovisioned here, and wasting my money.

But I am not sure why I seem so restricted by insufficient CPU. Is there some config I need to change? Any guidance would be appreciated.

Allocatable:
 cpu:                940m
 ephemeral-storage:  47093746742
 hugepages-2Mi:      0
 memory:             2702216Ki
 pods:               110
Non-terminated Pods:         (7 in total)
  Namespace                  Name                                                CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------                  ----                                                ------------  ----------  ---------------  -------------
  default                    mission-worker-5cf6654687-fwmk4                     100m (10%)    0 (0%)      0 (0%)           0 (0%)
  default                    mission-worker-5cf6654687-lnwkt                     100m (10%)    0 (0%)      0 (0%)           0 (0%)
  kube-system                fluentd-gcp-v3.1.1-5b6km                            100m (10%)    1 (106%)    200Mi (7%)       500Mi (18%)
  kube-system                kube-dns-76dbb796c5-jgljr                           260m (27%)    0 (0%)      110Mi (4%)       170Mi (6%)
  kube-system                kube-proxy-gke-test-cluster-pool-1-96c6d8b2-m15p    100m (10%)    0 (0%)      0 (0%)           0 (0%)
  kube-system                metadata-agent-nb4dp                                40m (4%)      0 (0%)      50Mi (1%)        0 (0%)
  kube-system                prometheus-to-sd-gwlkv                              1m (0%)       3m (0%)     20Mi (0%)        20Mi (0%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource  Requests     Limits
  --------  --------     ------
  cpu       701m (74%)   1003m (106%)
  memory    380Mi (14%)  690Mi (26%)
Events:     <none>

Perhaps you can add a sample of your deployment file. Do you have any resource limits configurations such as a cpu limit? Maybe you should try to set requests to the desired amount. Read more here: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ — Shani C, May 17 '19 at 18:25
you also can look at "kubectl describe node NODE_NAME", especially "Allocatable" resources. — Vasili Angapov, May 17 '19 at 19:07
This was part of solution: https://github.com/kubernetes/kubernetes/issues/33777#issuecomment-426487268 — atkayla, May 22 '19 at 03:25

score 4 · Accepted Answer · answered May 21 '19 at 09:17

After the deployment, check the node capacities with kubectl describe nodes. For e.g: In the code example at the bottom of the answer:

Allocatable cpu: 1800m

Already used by pods in kube-system namespace: 100m + 260m + +100m + 200m + 20m = 680m

Which means 1800m - 680m = 1120m is left for you to use

So, if your pod or pods request for more than 1120m cpu, they will not fit on this node

So in order to get 2 pods of app up total = CPU(cores): 10m, MEMORY: 200Mi, it requires another entire +1 node = 2 nodes = 2 vCPU, 7.5 GB to make it work. If I try to deploy those 2 pods on the same single node, I get insufficient CPU error.

If you do the exercise described above, you will find your answer. In case, there is enough cpu for your pods to use and still you are getting insufficient CPU error, check if you are setting the cpu request and limit params correctly. See here

If you do all the above and still it's an issue. Then, I think in your case, what could be happening is that you are allocating 5-10m cpu for your node app which is too less cpu to allocate. Try increasing that may be to 50m cpu.

I have a feeling I should actually be able to run a handful of pod replicas (like 3 replicas and more) on 1 node of f1-micro (1 vCPU, 0.6 GB) or f1-small (1 vCPU, 1.7 GB), and that I am way overprovisioned here, and wasting my money.

Again, do the exercise describe above to conclude that

Name:            e2e-test-minion-group-4lw4
[ ... lines removed for clarity ...]
Capacity:
 cpu:                               2
 memory:                            7679792Ki
 pods:                              110
Allocatable:
 cpu:                               1800m
 memory:                            7474992Ki
 pods:                              110
[ ... lines removed for clarity ...]
Non-terminated Pods:        (5 in total)
  Namespace    Name                                  CPU Requests  CPU Limits  Memory Requests  Memory Limits
  ---------    ----                                  ------------  ----------  ---------------  -------------
  kube-system  fluentd-gcp-v1.38-28bv1               100m (5%)     0 (0%)      200Mi (2%)       200Mi (2%)
  kube-system  kube-dns-3297075139-61lj3             260m (13%)    0 (0%)      100Mi (1%)       170Mi (2%)
  kube-system  kube-proxy-e2e-test-...               100m (5%)     0 (0%)      0 (0%)           0 (0%)
  kube-system  monitoring-influxdb-grafana-v4-z1m12  200m (10%)    200m (10%)  600Mi (8%)       600Mi (8%)
  kube-system  node-problem-detector-v0.1-fj7m3      20m (1%)      200m (10%)  20Mi (0%)        100Mi (1%)
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  CPU Requests    CPU Limits    Memory Requests    Memory Limits
  ------------    ----------    ---------------    -------------
  680m (34%)      400m (20%)    920Mi (12%)        1070Mi (14%)

This was part of solution: https://github.com/kubernetes/kubernetes/issues/33777#issuecomment-426487268 — atkayla, May 22 '19 at 03:25

GKE Insufficient CPU for small Node.js app pods

1 Answers1

Linked