Skaffold is not fully running all the services

Question

I'm developing this dummy project and trying to make it work locally via Skaffold.

There are 3 services in my project (running on ports 3001, 3002 and 3003 respectively), wired via NATS server.

The problem is: I get different kinds of errors each time I run skaffold debug, and one/more service(s) don't work.

At times, I don't get any errors, and all services work as expected. The followings are some of the errors:

Waited for <...>s due to client-side throttling, not priority and fairness,
request: GET:https://kubernetes.docker.internal:6443/api/v1/namespaces/default/pods?labelSelector=app%!D(MISSING). <...>%!C(MISSING)app.kubernetes.io%!F(MISSING)managed-by%!D(MISSING)skaffold%!C(MISSING)skaffold.dev%!F(MISSING)run-id%!D(MISSING)<...>` (from `request.go:668`)
- `0/1 nodes are available: 1 Insufficient cpu.` (from deployments)
- `UnhandledPromiseRejectionWarning: NatsError: CONNECTION_REFUSED` (from apps)
- `UnhandledPromiseRejectionWarning: Error: getaddrinfo EAI_AGAIN nats-service` (from apps)

I'm at a loss and can't help myself anymore. I hope someone here will be able to help me out.

Thanks in advance.

PS: Below is my machine's config, in case it's my machine's fault.

Processor: AMD Ryzen 7 1700 (8C/16T)
Memory: 2 x 8GB DDR4 2667MHz
Graphics: AMD Radeon RX 590 (8GB)
OS: Windows 10 Pro 21H1

$ docker version
Client:
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.12
 Git commit:        0ed913b8-
 Built:             07/28/2020 16:36:03
 OS/Arch:           windows/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:10 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:38:50Z", GoVersion:"go1.16.8", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.4", GitCommit:"3cce4a82b44f032d0cd1a1790e6d2f5a55d20aae", GitTreeState:"clean", BuildDate:"2021-08-11T18:10:22Z", GoVersion:"go1.16.7", Compiler:"gc", Platform:"linux/amd64"}

I use WSL2 (Debian) and docker-desktop is the context of Kubernetes.

0/1 nodes are available: 1 Insufficient cpu. (from deployments) This error clearly means your deployment doesn't have sufficient resources to startup. — Harsh Manvar, Sep 28 '21 at 19:10
Could you please paste output of the `kubernetes describe nodes` command? — Mikolaj S., Sep 29 '21 at 15:06
Thanks, based on that it seems that your cluster can handle this application. You mentioned under Harsh Manvar's answer that after restart it is working. Is it still working or do you have issues? — Mikolaj S., Oct 04 '21 at 07:10
@MikolajS. I still have issues. I don't know where the problem is. I tried my best to follow industry standards while building this dummy project, so let me know if there is any possibility to improve the layout and structure and all. It works with fresh startup of OS, and keeps crashing after 2-3 runs (like 2-3 restarts of the deployments). And I moved on with volumes since then, and fell into problem like https://stackoverflow.com/q/60774220. I'm still looking for solution of this, and now volume issue has been added too. — msrumon, Oct 05 '21 at 08:12
So afer 2-3 runs are you getting `0/1 nodes are available: 1 Insufficient cpu.` error message? Did you try deleting the deployment first and then apply again? — Mikolaj S., Oct 05 '21 at 11:06
[One issue - one question, this is how Stack forum works.](https://meta.stackexchange.com/questions/39223/one-post-with-multiple-questions-or-multiple-posts). For other issues, please create new questions. — Mikolaj S., Oct 05 '21 at 11:09
> Did you try deleting the deployment first and then apply again? = Yes I did. **Skadfold** automatically deletes old resources and re-create them. — msrumon, Oct 06 '21 at 12:15
Are you sure that everything is deleted? Seems like something is still consuming your CPU... When you have `0/1 nodes are available: 1 Insufficient cpu`, could you run `kubernetes describe nodes` command and paste output? — Mikolaj S., Oct 06 '21 at 15:26
@MikolajS. https://pastebin.com/YLp5jgnn. Described everything there. — msrumon, Oct 07 '21 at 09:16
Ok, thanks. Seems that your deployments are requesting too much resources. Take a look - your node has 16 CPUs available, while each `deployment-service-app` and `deployment-service-db` are requesting 4 CPUs. Then for only one service it is taking half of CPUso you can't start three at once (they need 24 CPUs). Did you consider lowering the requests value? Do they have to be so high? — Mikolaj S., Oct 08 '21 at 11:45
No they don't. I've only set the upper limit - as written [here](https://github.com/msrumon/microservice-template/blob/master/k8s/common/nats.yaml#L48-L51). Should I also try setting up the `requests` key and set the lower limit too? Or what about completely omitting it? I tried that one, and still same issue, plus IDE also yells that the config has no resource limits and hence can eat up resources. Don't know what to do. — msrumon, Oct 09 '21 at 20:59
Also feel free to help me on [this](https://stackoverflow.com/q/69516951/11317272). — msrumon, Oct 10 '21 at 16:24
Hi, that's my mistake, sorry for that. That's true, you are setting only the upper limits. However, it is also setting the lower limits. Please check my answer for more details. — Mikolaj S., Oct 11 '21 at 09:39

CodeWizard · Answer 1 · 2021-09-28T19:23:09.140

The error say it all:

- `0/1 nodes are available: 1 Insufficient cpu.` (from deployments)

You are requesting or deploying more apps than your node can handle.
In your deployment units (Deployment, StatefulSet, DeamoSet etc) you define the resources which you require for your application, for example in this pod:

apiVersion: v1
kind: Pod
metadata:
  name: frontend
spec:
  containers:
  - name: app
    image: images.my-company.example/app:v4
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"
  - name: log-aggregator
    image: images.my-company.example/log-aggregator:v6
    resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

This is the relevant section:

resources:
      requests:
        memory: "64Mi"
        cpu: "250m"
      limits:
        memory: "128Mi"
        cpu: "500m"

Check your node resources and increase your nodes or resources in order to be able to deploy more services.

score 1 · Answer 2 · answered Sep 28 '21 at 19:20

1

0/1 nodes are available: 1 Insufficient cpu. (from deployments)

This error clearly means your deployment doesn't have sufficient resources to startup.

i tried deploying the same on my environment same error, 2 CPU is not enough to start this image.

You might need a higher resource or big node to test

https://github.com/msrumon/microservice-template/blob/master/k8s/nats.yaml

answered Sep 28 '21 at 19:20

Harsh Manvar

27,020
6
48
102

I tried restarting my PC (hence minimum amount of blocked resources) - and tried to run `skaffold debug`. And guess what? All the services are up and running - without any hiccup. I think it's the issue of my PC config - maybe I need more CPU and memory. By the way, are you sure the resource limit should be increased for `nats` and not for anything else? If yes, how is it doing all fine after machine restart? – msrumon Sep 28 '21 at 19:27
it's it working fine no need to increase limit, but looks like it high resource-consuming app. – Harsh Manvar Sep 28 '21 at 19:28
Okay so what about the other errors? Have you got any clue of them? – msrumon Sep 28 '21 at 19:38
looks it's from app it self no able to connect to service maybe. – Harsh Manvar Sep 28 '21 at 19:39
I mean do you get the same error if you run it in your machine? I need to find out the issue, and if it's coming from the k8s configurations, I need a proper guideline about what would be the correct ones. – msrumon Sep 28 '21 at 19:47
with 2 CPU yes i faced the same error pod goes to the pending state with event insufficient cpu. – Harsh Manvar Sep 29 '21 at 03:34
Interesting! Is there any resource from where I can learn what should be the recommended resource limits? Or will I have to do trials after trials before I come up with optimal settings? – msrumon Sep 29 '21 at 04:03

score 1 · Accepted Answer · answered Oct 11 '21 at 09:37

The main reason of issues like this one is that you are setting only CPU limit (without setting CPU request) so Kubernetes automatically assigns a CPU request which is equal to the CPU limit:

If you specify a CPU limit for a Container but do not specify a CPU request, Kubernetes automatically assigns a CPU request that matches the limit. Similarly, if a Container specifies its own memory limit, but does not specify a memory request, Kubernetes automatically assigns a memory request that matches the limit.

So as requests are equal to the limits, your node can't meet these requirements (you have 16 CPUs available; to start all services you need 24 CPUs) - that's why you are getting 0/1 nodes are available: 1 Insufficient cpu error message.

How to fix it?

Set a CPU request which is different from the CPU limit
Delete CPU limits

But...

You wrote that:

Should I also try setting up the requests key and set the lower limit too? Or what about completely omitting it? I tried that one, and still same issue.

So if you deleted all CPU limits from all deployments and you still have error related to the insufficient CPU it clearly shows that your app is too resource-hungry. I'd suggest optimizing the application in terms of resource utilization. Another option is to increase node resources.

Yeah I both set `requests` and `limits` keys and it all went good. I accepted your answer. Thanks for your contribution. If you have time in hand, I have [this](https://stackoverflow.com/q/69516951/11317272) and [this](https://stackoverflow.com/q/69518264/11317272) issues with the same project. You're most welcome to contribute there as well. — msrumon, Oct 11 '21 at 11:13

Skaffold is not fully running all the services

3 Answers3

Linked