ingress-nginx, cert-manager and ingressClassName

Question

I recently upgraded ingress-nginx to version 1.0.3.

As a result, I removed the kubernetes.io/ingress.class annotation from my ingress, and put .spec.ingressClassName instead.

I am running cert-manager-v1.4.0.

This morning I had an email saying that my Let's Encrypt certificate will expire in 10 days. I tried to figure out what was wrong with it - not positive that it was entirely due to the ingress-nginx upgrade.

I deleted the CertificateRequest to see if it would fix itself. I got a new Ingress with the challenge, but:

1. The challenge ingress had the kubernetes.io/ingress.class annotation set correctly, even though my ingress has .spec.ingressClassName instead - don't know how or why, but it seems like it should be OK.

2. However, the challenge ingress wasn't picked up by the ingress controller, it said:

ingress class annotation is not equal to the expected by Ingress Controller

I guess it wants only the .spec.ingressClassName even though I thought the annotation was supposed to work as well.

So I manually set .spec.ingressClassName on the challenge ingress. It was immediately seen by the ingress controller, and the rest of the process ran smoothly, and I got a new cert - yay.

It seems to me like this will happen again, so I need to know how to either:

1. Convince cert-manager to create the challenge ingress with .spec.ingressClassName instead of kubernetes.io/ingress.class. Maybe this is fixed in 1.5 or 1.6?

2. Convince ingress-nginx to respect the kubernetes.io/ingress.class annotation for the challenge ingress. I don't know why this doesn't work.

Answer 1

Issue

The issue was fixed by certificate renewal, it works fine without manually set spec.ingressClassName in challenge ingress (I saw it with older version), issue was somewhere else.

Also with last available (at the writing moment) cert-manager v1.5.4 challenge ingress has the right setup "out of the box":

spec:
  ingressClassName: nginx
---
$ kubectl get ing
NAME                        CLASS    HOSTS            ADDRESS         PORTS     AGE
cm-acme-http-solver-szxfg   nginx    dummy-host       ip_address      80        11s

How it works (concept)

I'll describe main steps how this process works so troubleshooting will be straight-forward in almost all cases. I'll take a letsencypt staging as an issuer.

There's a chain when certificate is requested to be created which issuer follows to complete (all resources have owners - previous resource in chain):

main ingress resource -> certificate -> certificaterequest -> order -> challenge -> challenge ingress.

Knowing this, if something failed, you can go down by the chain and using kubectl describe command find where the issue appeared.

Troubleshooting example

I intentionally added a wrong domain in ingress to .spec.tls.hosts and applied it. Below how the chain will look like (all names will be unique!):

See certificates:

$ kubectl get cert
NAME                     READY   SECRET                          AGE
lets-secret-test-2       False   lets-secret-test-2              15m

Describe certificate we are interested in (you can notice I changed domain, there was already secret):

$ kubectl describe cert lets-secret-test-2
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Issuing    16m   cert-manager  Existing issued Secret is not up to date for spec: [spec.commonName spec.dnsNames]
  Normal  Reused     16m   cert-manager  Reusing private key stored in existing Secret resource "lets-secret-test-2"
  Normal  Requested  16m   cert-manager  Created new CertificateRequest resource "lets-secret-test-2-pvb25"

Nothing suspicious here, moving forward.

$ kubectl get certificaterequest
NAME                           APPROVED   DENIED   READY   ISSUER                REQUESTOR                                         AGE
lets-secret-test-2-pvb25       True                False   letsencrypt-staging   system:serviceaccount:cert-manager:cert-manager   19m

Describing certificaterequest:

$ kubectl describe certificaterequest lets-secret-test-2-pvb25
Events:
  Type    Reason           Age   From          Message
  ----    ------           ----  ----          -------
  Normal  cert-manager.io  19m   cert-manager  Certificate request has been approved by cert-manager.io
  Normal  OrderCreated     19m   cert-manager  Created Order resource default/lets-secret-test-2-pvb25-2336849393

Again, everything looks fine, no errors, moving forward to order:

$ kubectl get order
NAME                                  STATE     AGE
lets-secret-test-2-pvb25-2336849393   pending   21m

It says pending, that's closer:

$ kubectl describe order lets-secret-test-2-pvb25-2336849393

Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Created  21m   cert-manager  Created Challenge resource "lets-secret-test-2-pvb25-2336849393-3788447910" for domain "dummy-domain"

Challenge may shed some light, moving forward:

$ kubectl get challenge
NAME                                             STATE     DOMAIN           AGE
lets-secret-test-2-pvb25-2336849393-3788447910   pending   dummy-domain   23m

Describing it:

$ kubectl describe challenge lets-secret-test-2-pvb25-2336849393-3788447910

Checking status:

Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://dummy-domain/.well-known/acme-challenge/xxxxyyyyzzzzz': Get "http://dummy-domain/.well-known/acme-challenge/xxxxyyyyzzzzz": dial tcp: lookup dummy-domain on xx.yy.zz.ww:53: no such host
  State:       pending

Now it's clear that something is wrong with domain, worth checking it:

Found and fixed the "mistake":

$ kubectl apply -f ingress.yaml
ingress.networking.k8s.io/ingress configured

Certificate is ready!

$ kubectl get cert
NAME                     READY   SECRET                          AGE
lets-secret-test-2       True    lets-secret-test-2              26m

Correct way to renew a certificate using cert-manager

It's possible to renew a certificate by deleting corresponding secret, however documentation says it's not recommended:

Deleting the Secret resource associated with a Certificate resource is not a recommended solution for manually rotating the private key. The recommended way to manually rotate the private key is to trigger the reissuance of the Certificate resource with the following command (requires the kubectl cert-manager plugin):

kubectl cert-manager renew cert-1

Kubectl cert-manager command installation process is described here as well as other commands and examples.

Useful links:

• Certificate in cert-manager - concept
• Cert-manager - certificate request
• ACME Orders and Challenges