How to setup TLS correctly in Kubernetes via cert-manager?

Question

I'm trying to setup TLS for a service that's available outside a Kubernetes cluster (AWS EKS). With cert-manager, I've successfully issued a certificate and configured ingress, but I'm still getting error NET::ERR_CERT_AUTHORITY_INVALID. Here's what I have:

1. namespace tests and hello-kubernetes in it (both deployment and service have name hello-kubernetes-first, serivce is ClusterIP with port 80 and targetPort 8080, deployment is based on paulbouwer/hello-kubernetes:1.8, see details in my previous question)

2. DNS and ingress configured to show the service:

   apiVersion: networking.k8s.io/v1
   kind: Ingress
   metadata:
     name: hello-kubernetes-ingress
     namespace: tests
   spec:
     ingressClassName: nginx
     rules:
     - host: test3.projectname.org
       http:
         paths:
         - path: "/"
           pathType: Prefix
           backend:
             service:
               name: hello-kubernetes-first
               port:
                 number: 80
   

   Without configuring TLS, I can access test3.projectname.org via http and see the service (well, it tries to redirect me to https, I see NET::ERR_CERT_AUTHORITY_INVALID, I go to insecure anyway and see the hello-kubernetes page).

   • note: I have nginx-ingress ingress controller; it was installed before me via the following chart:

     apiVersion: v2
     name: nginx
     description: A Helm chart for Kubernetes
     type: application
     version: 4.0.6
     appVersion: "1.0.4"
     dependencies:
     - name: ingress-nginx
       version: 4.0.6
       repository: https://kubernetes.github.io/ingress-nginx
     

     and the values overwrites applied with the chart differ from the original ones mostly in extraArgs: default-ssl-certificate: "nginx-ingress/dragon-family-com" is uncommneted

3. cert-manager installed via kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.4/cert-manager.yaml

4. ClusterIssuer created with the following config:

   apiVersion: cert-manager.io/v1
   kind: ClusterIssuer
   metadata:
     name: letsencrypt-backoffice
   spec:
     acme:
       server: https://acme-staging-v02.api.letsencrypt.org/directory
       # use https://acme-v02.api.letsencrypt.org/directory after everything is fixed and works
       privateKeySecretRef: # this secret is created in the namespace of cert-manager
         name: letsencrypt-backoffice-private-key
       # email: <will be used for urgent alerts about expiration etc>
   
       solvers:
       # TODO: add for each domain/second-level domain/*.projectname.org
       - selector:
           dnsZones:
             - test.projectname.org
             - test2.projectname.org
             - test3.projectname.org
         http01:
           ingress:
             class: nginx
   

5. certificate in the tests namespace. It's config is

   apiVersion: cert-manager.io/v1
   kind: Certificate
   metadata:
     name: letsencrypt-certificate-31
     namespace: tests
   spec:
     secretName: tls-secret-31
     issuerRef:
       kind: ClusterIssuer
       name: letsencrypt-backoffice
     commonName: test3.projectname.org
     dnsNames:
     - test3.projectname.org
   

Now, certificate is ready (kubectl get certificates -n tests tells that) and to apply it, I add this to ingress's spec:

  tls:
    - hosts:
      - test3.projectname.org
      secretName: tls-secret-31

However, when I try to open test3.projectname.org via https, it still shows me the NET::ERR_CERT_AUTHORITY_INVALID error. What am I doing wrong? How to debug this? I've checked up openssl s_client -connect test3.projectname.org:443 -prexit* and it shows the following chain:

 0 s:CN = test3.projectname.org
   i:C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
 1 s:C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
   i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
 2 s:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
   i:C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Doctored Durian Root CA X3

and tells, among other output

Verification error: unable to get local issuer certificate

Unfortunately, I haven't found anything useful to try further, so any help is appreciated.

Answer 1

Your ClusterIssuer refers to LetsEncrypt staging issuer. Remove that setting / the default should use their production setup. As pointed out in comments: https://acme-v02.api.letsencrypt.org/directory

Deleting the previously generated secrets or switching to new secrets should ensure your certificates would be re-generated, using the right issuer.

The staging issuer could be useful testing LetsEncrypt integration, it shouldn't be used otherwise.

Answer 2

Following the suggestion from SYN, I've fixed this by

1. switching ACME server in ClusterIssuer config from https://acme-staging-v02.api.letsencrypt.org/directory to https://acme-v02.api.letsencrypt.org/directory. The idea of the staging server seems to be: allow to debug certificate issuing (so that kubectl get certificate [-n <namespace>] shows that READY = true) without providing actual trusted certificates; after certificate issuing is ok, one has to switch to the main server to get production certificates.

2. Updating certificates, tls secrets and ingress configs. Well, I'm not sure if there's a way to actually update certificates; instead, I've created new ones, which created new secrets, and then updated ingress configs (just secrets' names)

Answer 3

The reason that your certificates didn't work, it not because you used staging server, but because you didn't specify the tls object within the Ingress rules. Certbot's staging exists only for the purpose of testing, and for not "ban" you while you testing things out if you request more than 5 certificates/hour. When you verify that everything works as expected, you can use the normal non-staging server.

This is how it should be done:

Cluster Issuer object:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: user@example.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - http01:
        ingress:
          class: nginx

Ingress Object:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hello-kubernetes-ingress
  namespace: tests
  labels:
    app: hello-kubernetes-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  tls:
  - hosts:
    - test3.projectname.org
    secretName: hello-tls
  rules:
  - host: test3.projectname.org
    http:
      paths:
      - pathType: ImplementationSpecific
        path: "/"
        backend:
          service:
            name: hello-kubernetes-ingress
            port: 
              number: 80

The certificate and the key, are stored in a secret called "hello-tls" which you didn't also specify in your initial example, hence the failure you was receiving.