1

I'm setting up a k8s cluster on premise using kubespray.

I'm trying to harden the kubernetes cluster using CIS Benchmark documentation.

For the --kubelet-certificate-authority argument I set up the TLS connection between the apiserver and kubelets. Then, I edited the API server pod specification file /etc/kubernetes/manifests/kube-apiserver.yaml on the master node and set the --kubelet-certificate-authority parameter for the certificate authority like this : --kubelet-certificate-authority=/etc/kubernetes/ssl/apiserver.crt

But with that I'm not longer able top deploy pods (using helm), having the known error :

[centos@infra-vm ~]$ helm list Error: forwarding ports: error upgrading connection: error dialing backend: x509: cannot validate certificate for 192.168.33.143 because it doesn't contain any IP SANs

Where 192.168.33.143 is the master node IP address.

I've checked the above certificate autority and is has IP SANs : enter image description here

So I really can't figure out where the issue comes from .

[centos@infra-vm ~]$ kubectl get pod --namespace kube-system
NAME                                        READY   STATUS    RESTARTS   AGE
calico-kube-controllers-7555c9885c-tjz78    1/1     Running   0          3d21h
calico-node-2p4p4                           1/1     Running   0          3d21h
calico-node-4rhzj                           1/1     Running   0          3d21h
coredns-56bc6b976d-wrxsl                    1/1     Running   0          3d21h
coredns-56bc6b976d-zlvxb                    1/1     Running   0          3d21h
dns-autoscaler-5fc5fdbf6-sl6gg              1/1     Running   0          3d21h
kube-apiserver-cpu-node0                    1/1     Running   0          3d21h
kube-controller-manager-cpu-node0           1/1     Running   0          3d21h
nvidia-device-plugin-daemonset-1.12-zj82x   1/1     Running   0          3d20h
tiller-deploy-677fbf76bb-hcgtw              1/1     Running   0          3d21h

[centos@infra-vm ~]$ kubectl logs tiller-deploy-677fbf76bb-hcgtw  --namespace kube-system
Error from server: Get https://192.168.33.143:10250/containerLogs/kube-system/tiller-deploy-677fbf76bb-hcgtw/tiller: x509: cannot validate certificate for 192.168.33.143 because it doesn't contain any IP SANs
[centos@infra-vm ~]$

Could one try to help figure out what is going on?

nixmind
  • 2,060
  • 6
  • 32
  • 54

1 Answers1

1

First of all /etc/kubernetes/ssl/apiserver.crt is not a valid CA certificate. CA would have:

    X509v3 extensions:
        X509v3 Key Usage: critical
            Digital Signature, Key Encipherment, Certificate Sign

Notice Certificate Sign extension that allows for signing certificates.

You are seeing this error: cannot validate certificate for 192.168.33.143 because it doesn't contain any IP SANs because kubelet is using self signed certificates to serve https traffic on port 10250 and you are using invalid certificate to validate it.

So what should you do to make it work??

  • Use /etc/kubernetes/ssl/ca.crt to sign new certificate for kubelet with valid IP SANs.
  • Set --kubelet-certificate-authority=/etc/kubernetes/ssl/ca.crt (valid CA).
  • In /var/lib/kubelet/config.yaml (kubelet config file) set tlsCertFile and tlsPrivateKeyFile to point to newly created kubelet crt and key files.
Matt
  • 7,419
  • 1
  • 11
  • 22
  • I fisrt used `/etc/kubernetes/ssl/ca.crt` and verify it before but got the same error, that's why I used `/etc/kubernetes/ssl/apiserver.crt`. And yes of course `ca.crt` has `Certificate Sign` extension but not IP SANs. So if you thing I should use `ca.crt` do you mean the problem comes from the above mentioned `/var/lib/kubelet/config.yaml` file? – nixmind Sep 23 '20 at 11:05
  • Sorry @matt, I didn't read well your answer, you asked to use `ca.crt` to sign new certificate for kubelet with valid IP SANS. – nixmind Sep 23 '20 at 13:15
  • I connected to each of my node and run `kubelet` command and got this error message everytime : `[centos@cpu-node1 ~]$ kubelet F0928 10:33:56.371870 25519 server.go:249] error reading /var/lib/kubelet/pki/kubelet.key, certificate and key must be supplied as a pair `. In the `/etc/kubernetes/kubelet.conf` file I can see a cluster named `default-cluster` with a `certificate-authority-data` parameter and a user named `default-auth` with parameters `client-certificate` and `client-key`, both set to : `/var/lib/kubelet/pki/kubelet-client-current.pem` – nixmind Sep 28 '20 at 11:30
  • 1
    Please where do I stand to properly setup the TLS encryption between kubelet and apiserver for hardening purposes as stated above? It's very confusing from the k8s documentation, and in my case I'm working on a kubespray set up. I'd really appreciate some precise explanation. – nixmind Sep 28 '20 at 11:33
  • When I try to load the certificate authority data pricked form the `kubelet.conf` file, I've this error : `unable to load certificate 140359407167376:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:707:Expecting: TRUSTED CERTIFICATE` – nixmind Sep 28 '20 at 11:42
  • I don't have enough knowledge about how kubespray is running k8s cluster so I cannot give you step by step instructions. I have already told you what to change and the instructions I gave you were working for me so that is all I can do. – Matt Sep 28 '20 at 12:30
  • I don't really ask a step by step how to, please just let me know where in the process/ansible playbooks you apply the above suggested steps. – nixmind Sep 29 '20 at 19:12
  • I did not apply this in playbooks. I did it by hand. I dont know ansible – Matt Sep 30 '20 at 06:09
  • OK thx @Matt.. Please just let me know what you did by hand and where you put it on your infrastructure. What did you do on the master node? What did you do on the other nodes? Or did you put all certificates on the master node under `/etc/kubernetes/pki/' folder? How did you sign the new certificate with the `ca.crt` with valid IP SANs as mentioned above in your answer, and where did you put the signed certificate. Thank you so much for your attention – nixmind Sep 30 '20 at 09:49
  • Implementing the above @Matt procedure through ansible solved my issue ;) – nixmind Oct 19 '20 at 07:13