2

I've recently upgraded with kubeadm, which I expect to rotate all certificates, and for good measure, I also ran kubeadm init phase certs all, but I'm not sure what steps are required to verify that the certs are all properly in place and not about to expire.

I've seen a SO answer reference kubeadm init phase kubeconfig all is required in addition, but cannot find in the kubernetes kubeadm documentation telling me that it needs to be used in conjunction with phase certs.

What do I need to do to make sure that the cluster will not encounter expired certificates?

I've tried verifying by connecting to the secure local port: echo -n | openssl s_client -connect localhost:10250 2>&1 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | openssl x509 -text -noout | grep Not, which gives me expirations next month.

While openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text and openssl x509 -in /etc/kubernetes/pki/apiserver-kubelet-client.crt -noout -text yield dates over a year in advance.

These conflicting dates certainly have me concerned that I will find myself like many others with expired certificates. How do I get in front of that?

Thank you for any guidance.

Rico
  • 58,485
  • 12
  • 111
  • 141
Dan Bowling
  • 1,205
  • 1
  • 15
  • 41

2 Answers2

3

In essence kubeadm init phase certs all regenerates all your certificates including your ca.crt (Certificate Authority), and Kubernetes components use certificate-based authentication to connect to the kube-apiserver (kubelet, kube-scheduler, kube-controller-manager) so you will also have to regenerate pretty much all of those configs by running kubeadm init phase kubeconfig all

Keep in mind that you will have to regenerate the kubelet.conf on all your nodes since it also needs to be updated to connect to the kube-apiserver with the new ca.crt. Also, make sure you add all your hostnames/IP addresses that your kube-apiserver is going to serve on to the kubeadm init phase certs all command (--apiserver-cert-extra-sans)

Most likely you are not seeing the updated certs when connecting through openssl is because you haven't restarted the Kubernetes components and in particular the kube-apiserver. So you will have to start your kube-apiserver, kube-scheduler, kube-controller-manager, etc (or kube-apiservers, kube-schedulers, etc if you are running a multi-master control plane) You will also have to restart your kubelets on all your nodes.

Rico
  • 58,485
  • 12
  • 111
  • 141
  • Thank you for your explanation of what is going on, though after deleting all .conf and files in pki, `phase certs` and `phase kubeconfig` plus a reboot of the server did not change the `openssl` certificate expiry indication. – Dan Bowling Jun 03 '19 at 03:02
  • Did you reboot all your k8s masters? – Rico Jun 03 '19 at 05:38
  • yes. It's a single master that has the taint removed, no worker nodes. – Dan Bowling Jun 03 '19 at 14:49
  • It may be configured to get the certificates from another place. Are your configs under `/etc/kubernetes/...`? – Rico Jun 03 '19 at 15:13
  • Yes, our configs are under `/etc/kubernetes`. I've verified by looking at the systemd kubelet service. – Dan Bowling Jun 04 '19 at 21:43
  • That's super weird. Unless your certs are embedded in the container of the given k8s component. For example, in the kube-apiserver container. – Rico Jun 04 '19 at 21:48
0

A month later, I've learned a little more and wanted to update this question for those who follow behind me.

I filed an issue on Kubernetes requesting more information on how the kubeadm upgrade process automatically updates certificates. The documentation on Kubernetes says:

Note: kubelet.conf is not included in the list above because kubeadm configures kubelet for automatic certificate renewal.

After upgrading, I did not see an automatic cert renewal for the kubelet. I was then informed that:

the decision on when to rotate the certificate is non-deterministic and it may happen 70 - 90% of the total lifespan of the certificate to prevent overlap on node cert rotations.

They also provided the following process, which resolved my last outstanding certificate rotation:

sudo mv /var/lib/kubelet/pki /var/lib/kubelet/pki-backup
sudo systemctl restart kubelet
# the pki folder should be re-created.
Dan Bowling
  • 1,205
  • 1
  • 15
  • 41