0

For some reason my certificates cannot be applied to my k8s cluster. I can see that general traffic flow is running, ie. using http my site is up and running.

I'm using: kubernetes 1.22 cert-manager 1.6.1

My ingress file looks like this:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web
  namespace: web
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt-production"
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.org/websocket-services: web
    nginx.ingress.kubernetes.io/websocket-services: web
    nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - some.example.com
      secretName: example-tls
  rules:
    - host: some.example.com
      http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: web
                port:
                  number: 80

Clusterissuer (letsencrypt-production) file is:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-production
spec:
  acme:
    email: services@chimeraprime.com
    privateKeySecretRef:
      name: letsencrypt-production
    server: https://acme-v02.api.letsencrypt.org/directory
    solvers:
    - http01:
        ingress:
          class: nginx
status:
  acme:
    lastRegisteredEmail: email@mycompany.com
    uri: https://acme-v02.api.letsencrypt.org/acme/acct/355122560
  conditions:
  - lastTransitionTime: "2022-01-08T17:45:55Z"
    message: The ACME account was registered with the ACME server
    observedGeneration: 1
    reason: ACMEAccountRegistered
    status: "True"
    type: Ready

Order is in pending state. Below the info from kubectl describe:

Name:         web-web-tls-h4pn7-1463892238
Namespace:    web
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: web-web-tls
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: web-web-tls-pnrb9
API Version:  acme.cert-manager.io/v1
Kind:         Order
Metadata:
  Creation Timestamp:  2022-01-10T21:20:47Z
  Generation:          1
    Manager:      controller
    Operation:    Update
    Time:         2022-01-10T21:20:47Z
    API Version:  acme.cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:authorizations:
        f:finalizeURL:
        f:state:
        f:url:
    Manager:      controller
    Operation:    Update
    Subresource:  status
    Time:         2022-01-10T21:20:47Z
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  web-web-tls-h4pn7
    UID:                   356d2130-bb03-4cba-a751-cff5904b331c
  Resource Version:        32432743
  UID:                     7ae44312-9565-4656-bc71-6a921f8d899f
Spec:
  Dns Names:
    some.example.com
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt-production
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ2d6Q0NBV3NDQVFBd0FEQ0NBU0l3RFFZSktvWklodmNOQVFFQkJRQURnZ0VQQURDQ0FRb0NnZ0VCQU9CYwppZGVwS3dibkpQQWJhbnB0M0o4ZXZINisxcmd4N2JPbjdkQm1nR0k1L2QzOFpqVEFuWFRuNkg0QmJyNWxzTWxmCjZpT1MxT1EyWExORFh2Zlc5eENLbmdUbVB2aFdTVTZVMHppSE11bm95RDFIU2JaK3VlZXR3U1VrdDZYaGhCVXUKOC92L1FvU04zQm9OaHNqWjdMSThDMUt5RVU2K1d0TGt6RGNWbHZHdG4wSTNsVXhhTE9tYUEvRGtxYUdEcENMZQpKZmFoS1l4QmdiK004TndrVzVPc3F0T1QydWFMRzBmTVBMWDdJNTlMQXgwSnNsOHg3VzlNZTFZemdYVE5GcTZsCkM1dURpM2x5Z2xZNXdSZTExU1ZBdWpwV2FUWU1pQnl5T1Vway8vRmhIOG43U1B1c2FDaXRTdUV3S0VBU1FDVVIKY1h5T3FnUXUwNkIwT1BCaVBDOENBd0VBQWFBK01Ed0dDU3FHU0liM0RRRUpEakV2TUMwd0hnWURWUjBSQkJjdwpGWUlUWm1sdWEyNHVZMmhwYldWeVlYQnlhUzV0WlRBTEJnTlZIUThFQkFNQ0JhQXdEUVlKS29aSWh2Y05BUUVMCkJRQURnZ0VCQU5ON2VReTd0ZDgySWZoNUJuODBjRGFMcEN5WU9tSjd4RzlQUHJEaEl2RTlLTS8zdzN0WXJFem4KRXIraFczSU15WnY4MnhpYzRhbnFYclBIdEpzcjd4M3ZxVktNcVM5QUhmbGhMVGV4VUhCL0lRTWtkVG5tM3JaOAp3QUpmQUVBY21FU2hQNVpmYWJ3NEJZcWluc2U3Vk5KVWY2Sm94M1dKTGdsUnJSVlJrRVJ5SkR6Vy9oTWEzL085CjZGWkh1eTFGUENQUGRpM0lkZ2wwT0E4ek5yU0IyelloRWp5Q0hId3FPSEFDOEMxeU1vamJQeG5xeThEMlovVlAKWjd6dWlzTHZwU1lzWVdEYldnd3NwTmFHQ09ZM01QZmxtUDFJdzBGOEtlN3BIek1uaHRNUGFwUTV3QWw0SklCOQpJOVpObXBWam5HRXIra3VIYjFWWGhNTGRJcjhyUEZVPQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KK
Status:
  Authorizations:
    Challenges:
      Token:        MA2X6cC5s4KehiEhNPANFDAEgjzHTgDlh5JVjvqjJ8U
      Type:         http-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/66966354360/HF2Iag
      Token:        MA2X6cC5s4KehiEhNPANFDAEgjzHTgDlh5JVjvqjJ8U
      Type:         dns-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/66966354360/sifuZA
      Token:        MA2X6cC5s4KehiEhNPANFDAEgjzHTgDlh5JVjvqjJ8U
      Type:         tls-alpn-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/66966354360/NuJ8Yw
    Identifier:     some.example.com
    Initial State:  pending
    URL:            https://acme-v02.api.letsencrypt.org/acme/authz-v3/66966354360
    Wildcard:       false
  Finalize URL:     https://acme-v02.api.letsencrypt.org/acme/finalize/355122560/54310290910
  State:            pending
  URL:              https://acme-v02.api.letsencrypt.org/acme/order/355122560/54310290910
Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Created  29m   cert-manager  Created Challenge resource "web-web-tls-h4pn7-1463892238-998650753" for domain "finkn.chimerapri.me"

Created challenge has no state:

Name:         web-web-tls-h4pn7-1463892238-998650753
Namespace:    web
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
  Creation Timestamp:  2022-01-10T21:20:47Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
    Manager:    controller
    Operation:  Update
    Time:       2022-01-10T21:20:47Z
  Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  web-web-tls-h4pn7-1463892238
    UID:                   7ae44312-9565-4656-bc71-6a921f8d899g
  Resource Version:        32432749
  UID:                     559a5ce8-d181-423a-9706-6e7532c433ef
Spec:
  Authorization URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/66966354360
  Dns Name:           some.example.com
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt-production
  Key:      MA2X6cC5s4KehiEhNPANFDAEgjzHTgDlh5JVjvqjJ8U.ZeoVv0hyPHZ3wO-p2vQVZWEvuU3Ti8DQSsrUIGlwP1d
  Solver:
    http01:
      Ingress:
        Class:  nginx
  Token:        MA2X6cC5s4KehiEhNPANFDAEgjzHTgDlh5JVjvqjJ8W
  Type:         HTTP-01
  URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/66966354360/HF2Iag
  Wildcard:     false
Events:         <none>

What am I missing here, I've recently upgraded cert-manager from 1.0.1 to 1.6.1 and from then on I'm seeing this kind of issues.

[EDIT] I can see no logs related to this site in cert manager pods logs.

Murakami
  • 3,474
  • 7
  • 35
  • 89
  • Hi, does the "some.example.com" (based on your yamls) resolve to a globally available IP address? Is the ingress listening on port 443 on that IP address? – Piotr Malec Jan 11 '22 at 15:55
  • Yes, some.example.com resolves to globally available public IP in front of the cluster. What do you mean by ingress listening on port 443? – Murakami Jan 11 '22 at 21:54
  • I mean does your TLS termination work? Can you access the site via HTTPS from outside of the cluster (and get the message about missing certificate)? This kind of issue was described in a similar SO post: https://stackoverflow.com/q/63432101/12014434 . – Piotr Malec Jan 12 '22 at 15:06
  • So I think the problem with that was that I had some DNS not switched to target IP. Cert manager won’t result in ‚true’ state if there is at least one host cannot be resolved. Then the REST can’t be used. – Murakami Jan 21 '22 at 19:35

0 Answers0