k0s kubectl exec and kubectl port-forwarding are broken

Question

I have a simple nginx pod and a k0s cluster setup with the k0s binary. Now i want to connect to that pod, but i get this error:

$ kubectl port-forward frontend-deployment-786ddcb47-p5kkv 7000:80

error: error upgrading connection: error dialing backend: rpc error: code = Unavailable 
desc = connection error: desc = "transport: Error while dialing dial unix /var/lib/k0s/run/konnectivity-server/konnectivity-server.sock: connect: connection refused"

I dont understand why this happens and why it is tries to access /var/lib/k0s/run/konnectivity-server/konnectivity-server.sock which does not exist on my maschine.

Do I have to add my local dev maschine with k0s to the cluster?

Extract from pod describe:

Containers:
  frontend:
    Container ID:   containerd://897a8911cd31c6d58aef4b22da19dc8166cb7de713a7838bc1e486e497e9f1b2
    Image:          nginx:1.16
    Image ID:       docker.io/library/nginx@sha256:d20aa6d1cae56fd17cd458f4807e0de462caf2336f0b70b5eeb69fcaaf30dd9c
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 28 Jan 2021 14:20:58 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  3m43s  default-scheduler  Successfully assigned remove-me/frontend-deployment-786ddcb47-p5kkv to k0s-worker-2
  Normal  Pulling    3m42s  kubelet            Pulling image "nginx:1.16"
  Normal  Pulled     3m33s  kubelet            Successfully pulled image "nginx:1.16" in 9.702313183s
  Normal  Created    3m32s  kubelet            Created container frontend
  Normal  Started    3m32s  kubelet            Started container frontend

deployment.yml and service.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: frontend-deployment
  labels:
    app: frontend
spec:
  replicas: 2
  selector:
    matchLabels:
      app: frontend
  template:
    metadata:
      labels:
        app: frontend
    spec:
      containers:
      - name: frontend
        image: nginx:1.16
        ports:
        - containerPort: 80
----
apiVersion: v1
kind: Service
metadata:
  name: frontend-service
spec:
  selector:
    app: frontend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

can you provide the yamls? for solve your issue we need to reproduce the issue — Emon46, Jan 28 '21 at 13:48
added deployment and service yml. Service is probably unrelated, but that is all that is running in the name space. — TecBeast, Jan 29 '21 at 08:00

score 1 · Answer 1 · answered Feb 02 '21 at 13:05

1

Workaround is to just remove the file.

/var/lib/k0s/run/konnectivity-server/konnectivity-server.sock and restart the server.

Currenlty my github issue is still open.

https://github.com/k0sproject/k0s/issues/665

answered Feb 02 '21 at 13:05

TecBeast

930
8
16

score 0 · Answer 2 · edited Aug 29 '22 at 11:13

I had a similar issue where /var/lib/k0s/run/konnectivity-server/konnectivity-server.sock was not getting created.(in my case path was /run/k0s/konnectivity-server/konnectivity-server.sock )

I changed my host configuration which finally fixed this issue.

Hostname: The hostnames in my nodes were in uppercase but k0s somehow expected it to be in lower case. We can override the hostname in the configuration file but that still did not fix the konnectivity sock issue so I had to reset all my hostnames on nodes to small case hostnames.

This change finally fixed my issue and on my first attempt, I saw the .sock file was getting created but still didn't have the permission. Then I followed the suggestion given above by TecBeast and that fixed the issue permanently.

This issue was then raised as an Issue in the K0s GitHub repository, and fixed in a Pull Request.

k0s kubectl exec and kubectl port-forwarding are broken

2 Answers2