4

I'm following the Kubernetes install instructions for Helm: https://docs.cert-manager.io/en/latest/getting-started/install/kubernetes.html With Cert-manager v0.81 on K8 v1.15, Ubuntu 18.04 on-premise. When I get to testing the installation, I get these errors:

error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "issuers.admission.certmanager.k8s.io": the server is currently unable to handle the request
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "certificates.admission.certmanager.k8s.io": the server is currently unable to handle the request

If I apply the test-resources.yaml before installing with Helm, I'm not getting the errors but it is still not working. These errors are new to me, as Cert-manager used to work for me on my previous install about a month ago, following the same installation instructions. I've tried with Cert-Manager 0.72(CRD 0.7) as well as I think that was the last version I managed to get installed but its not working either.

What does these errors mean?

Update: It turned out to be an internal CoreDNS issue on my cluster. Somehow not being configured correctly. Possible related to wrong POD_CIDR configuration.

Isaack Rasmussen
  • 457
  • 4
  • 11
  • 1
    could you post your solution as an answer and accepted it? It will make your solution more visible if anyone will be searching for similar issues. – PjoterS Jul 05 '19 at 13:38

2 Answers2

3

If you experience this problem, check the logs of CoreDNS(Or KubeDNS) and you may see lots of errors related to contacting services. Unfortunately, I no longer have the errors. But this is how I figured out that my network setup was invalid.

I'm using Calico(Will apply for other networks as well) and its network was not set to the same as the POD_CIDR network that I initialized my Kubernetes with.

Example 1. Set up K8: 

kubeadm init --pod-network-cidr=10.244.0.0/16

  1. Configure Calico.yaml: 

    - name: CALICO_IPV4POOL_CIDR
  value: "10.244.0.0/16"
Isaack Rasmussen
  • 457
  • 4
  • 11
  • This is absolutely the answer. I realized this was an issue that pointed to a more fundamental problem with my configuration of the `cni`. And like you I solved this a few months ago, so I'm here to confirm that this is the solution. – L. J. Dec 29 '19 at 03:48
2

I also tried cert-manager v0.8.0 a very similar setup on Ubuntu 18.04 and k8s v1.14.1 and I began to get the same error when i tore down cert-manager using kubectl delete and reinstalled it, after experiencing some network issues on the cluster.

I stumbled on a solution that worked. On the master node, simply restart the apiserver container:

$ sudo docker ps -a | grep apiserver
af99f816c7ec        gcr.io/google_containers/kube-apiserver@sha256:53b987e5a2932bdaff88497081b488e3b56af5b6a14891895b08703129477d85               "/bin/sh -c '/usr/loc"   15 months ago       Up 19 hours                                     k8s_kube-apiserver_kube-apiserver-ip-xxxxxc_0
40f3a18050c3        gcr.io/google_containers/pause-amd64:3.0                                                                                      "/pause"                 15 months ago       Up 15 months                                    k8s_POD_kube-apiserver-ip-xxxc_0
$ sudo docker restart af99f816c7ec
af99f816c7ec
$

Then try applying the test-resources.yaml again:

$ kubectl apply -f test-resources.yaml
namespace/cert-manager-test unchanged
issuer.certmanager.k8s.io/test-selfsigned created
certificate.certmanager.k8s.io/selfsigned-cert created

If that does not work, this github issue mentions that the master node might need firewall rules to be able to reach the cert-manager-webhook pod. The exact steps to do so will depend on which cloud platform you are on.

L. J.
  • 136
  • 1
  • 4
  • Thanks for your reply, I finally got a chance to try it out. Unfortunately, restarting the API server did not help. Also, I'm on my own servers and there shouldn't be any firewall issues (currently disabled between the nodes) I've only got it to work by disabling webhooks on the helm chart but I do not know what consequence that has. – Isaack Rasmussen Jul 04 '19 at 16:00