I upgraded a Kubernetes cluster from version 1.22 to 1.25, after upgrading everything seems to run correctly. But on using kubectl exec or kubectl logs, I get the error: Error from server: error dialing backend: EOF.
I cannot find much about this error, it also seems to be an issue for the gitlab-runner, which tries to access the logs of its concurrent pods but is not able to.
Does anybody have a clue on where I could start? Our cluster runs on a private network and firewalls between the nodes are wide open, so this should not be an issue, and I can list all pods running on all nodes so the issue does not seem to lie in the network department.
One more caveat: I can get logs from pods on the master node, but not from pods on the agent nodes.
Exact output kubectl logs:
kubectl logs --namespace=<namespace> <pod name>
Error from server: Get "https://172.16.2.5:10250/containerLogs/<namespace>/<container>/<name>": EOF
