Issuing certificate as Secret does not exist

Question

Below is the describe output for both my clusterissuer and certificate reource. I am brand new to cert-manager so not 100% sure this is set up properly - we need to use http01 validation however we are not using an nginx controller. Right now we only have 2 microservices so the public-facing IP address simply belongs to a k8s service (type loadbalancer) which routes traffic to a pod where an Extensible Service Proxy container sits in front of the container running the application code. Using this set up I haven't been able to get anything beyond the errors below, however as I mentioned I'm brand new to cert-manager & ESP so this could be configured incorrectly...

Name:         clusterissuer-dev
Namespace:    
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
API Version:  cert-manager.io/v1beta1
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2020-08-07T18:46:29Z
  Generation:          1
  Resource Version:    4550439
  Self Link:           /apis/cert-manager.io/v1beta1/clusterissuers/clusterissuer-dev
  UID:                 65933d87-1893-49af-b90e-172919a18534
Spec:
  Acme:
    Email:  email@test.com
    Private Key Secret Ref:
      Name:  letsencrypt-dev
    Server:  https://acme-staging-v02.api.letsencrypt.org/directory
    Solvers:
      http01:
        Ingress:
          Class:  nginx
Status:
  Acme:
    Last Registered Email:  email@test.com
    Uri:                    https://acme-staging-v02.api.letsencrypt.org/acme/acct/15057658
  Conditions:
    Last Transition Time:  2020-08-07T18:46:30Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>


Name:         test-cert-default-ns
Namespace:    default
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
API Version:  cert-manager.io/v1beta1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2020-08-10T15:05:31Z
  Generation:          2
  Resource Version:    5961064
  Self Link:           /apis/cert-manager.io/v1beta1/namespaces/default/certificates/test-cert-default-ns
  UID:                 259f62e0-b272-47d6-b70e-dbcb7b4ed21b
Spec:
  Dns Names:
    dev.test.com
  Issuer Ref:
    Name:       clusterissuer-dev
  Secret Name:  clusterissuer-dev-tls
Status:
  Conditions:
    Last Transition Time:        2020-08-10T15:05:31Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2020-08-10T15:05:31Z
    Message:                     Issuing certificate as Secret does not exist
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  test-cert-default-ns-rrl7j
Events:
  Type    Reason     Age    From          Message
  ----    ------     ----   ----          -------
  Normal  Requested  2m51s  cert-manager  Created new CertificateRequest resource "test-cert-default-ns-c4wxd"

One last item - if I run the command kubectl get certificate -o wide I get the following output.

  NAME                           READY   SECRET                         ISSUER                     STATUS                                         AGE
  test-cert-default-ns           False   clusterissuer-dev-tls          clusterissuer-dev          Issuing certificate as Secret does not exist   2d23h

I used the following command: `kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.16.1/cert-manager.yaml` — NealR, Aug 11 '20 at 14:16
I also have same isue... I installed it with helm and can't figure it out — Djordje Nikolic, Aug 11 '20 at 20:11
Ok , I am in the same situation as you , are you installed cert-manager on a Baremetal kubernetes ? Can you follow these link : https://cert-manager.io/docs/faq/troubleshooting/ and https://cert-manager.io/docs/faq/acme/ . See where it block , for me it's on the challenge part , as cert-manager can't acces the port 80 of my server , I am going to work on that tomorow , tell me if you have the same problem ! — Popopame, Aug 12 '20 at 17:05
Can you please share the logs from `cert-manager` controller pod? Thanks. — hilsenrat, Aug 14 '20 at 17:53
Did you create the `clusterrolebinding` as mentioned in [this](https://cert-manager.io/docs/installation/kubernetes/) page? — Mr.KoopaKiller, Aug 17 '20 at 07:49
I resolved the problem , For me it was one of the ACME Challenges that was causing trouble , can you follow his link : https://cert-manager.io/docs/faq/troubleshooting/ . It is likely that the certificate is not creating because there is an error somewhere in the process , please follow the instruction ,and give us the precise error :) — Popopame, Aug 19 '20 at 06:14
This youtube video was pretty helpful in triaging this for me - https://www.youtube.com/watch?v=BlzRx6ROiX0&ab_channel=carpie.net. Walks thru the entire process. — slm, Feb 25 '21 at 02:40

sebisnow · Accepted Answer · 2021-02-01T12:57:38.240

I had the same issue and I followed the advice given in the comments by @Popopame suggesting to check out the troubleshooting guide of cert-manager to find out how to troubleshoot cert-manager. or [cert-managers troubleshooting guide for acme issues] to find out which part of the acme process breaks the setup.

It seems that often it is the acme-challenge where letsencrypt verifies the domain ownership by requesting a certain code be offered at port 80 at a certain path. For example: http://example.com/.well-known/acme-challenge/M8iYs4tG6gM-B8NHuraXRL31oRtcE4MtUxRFuH8qJmY. Notice the http:// that shows letsencrypt will try to validate domain ownership on port 80 of your desired domain.

So one of the common errors is, that cert-manager could not put the correct challenge in the correct path behind port 80. For example due to a firewall blocking port 80 on a bare metal server or a loadbalancer that only forwards port 443 to the kubernetes cluster and redirects to 443 directly.

Also be aware of the fact, that cert-manager tries to validate the ACME challenge as well so you should configure the firewalls to allow requests coming from your servers as well.

If you have trouble getting your certificate to a different namespace, this would be a good point to start with.

In your specific case I would guess at a problem with the ACME challenge as the CSR (Certificate Signing Request) was created as indicated in the bottom most describe line but nothing else happened.

This was essentially the issue. I had my ingress set to enforce https, which was preventing validation. Setting that flag to `false`, initially, fixed the problem. — NealR, Mar 08 '21 at 20:39
I still don't understand how you fixed the problem. can you show code? — Duru Cynthia Udoka, Aug 04 '22 at 11:16
The "Code" depends on your firewall solution, webserver that should respond to the ACME challenge or a reverse proxy in front of the k8s cluster. So check whether port 80 is allowed if not open it. Check if your reverse proxy into your k8s cluster (if you use one) forwards port 80 to the k8s ingress. Then check if you webserver / ingress accepts connections for port 80 and answers the [ACME challenge](https://letsencrypt.org/docs/challenge-types/). You can manually check by opening a browser at http:///.well-known/acme-challenge/ — sebisnow, Aug 12 '22 at 15:02

slm · Answer 2 · 2023-03-13T03:14:20.897

1. Setup Using Helm

By far the easiest method I've found was to use helm v3 to install cert-manager. I was able to set it up on a k3s cluster as follows:

$   helm repo add jetstack https://charts.jetstack.io
$   helm repo update
$   helm install \
        cert-manager jetstack/cert-manager \
        --namespace cert-manager \
        --version v1.2.0 \
        --create-namespace \
        --set installCRDs=true

2. Setup ClusterIssuer

Once it's installed you need to create a ClusterIssuer which can then be used when requesting certificates from let's encrypt.

$ more cert-clusterissuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-stg
spec:
  acme:
    email: my_letsencrypt_email@mydom.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: le-issuer-acct-key
    solvers:
    - dns01:
        cloudflare:
          email: my_cloudflare_email@mydom.com
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token
      selector:
        dnsZones:
        - 'mydomdom.org'
        - '*.mydomdom.org'

Deploy that, notice it'll get deployed into the same namespaces as cert-manager:

$ kubectl apply -f cert-clusterissuer.yaml

$ kubectl get clusterissuers
NAME              READY   AGE
letsencrypt-stg   True    53m

3. Setup Cloudflare API Token Secret

Deploy your Cloudflare API token into a secret and put it into the cert-manager namespace:

$ more cloudflare-api-token.yaml
apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque
stringData:
  api-token: <my cloudflare api token key>

$ kubectl apply -f cloudflare-api-token.yaml

4. Create a test Certificate

Now attempt to request the generation of a certificate from let's encrypt:

$ more test-certificate.yaml
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: le-test-mydomdom-org
  namespace: cert-manager
spec:
  secretName: le-test-mydomdom-org
  issuerRef:
    name: letsencrypt-stg
    kind: ClusterIssuer
  commonName: 'le-test.mydomdom.org'
  dnsNames:
  - "le-test.mydomdom.org"

$ kubectl -n cert-manager apply -f test-certificate.yaml

5. Debugging Certificate Creation

You can then watch the request as it flows through the various stages. I believe the flow is certificates -> certificaterequests -> orders -> challenges.

NOTE: Knowing this general flow was hugely helpful for me in terms of understanding where a request was failing within kubernetes as I was attempting to debug it.

When debugging you'll typically want to do kubectl get -n cert-manager <stage> -A to see a list of all the outstanding resources within that stage. Keep in mind that after a challenge is fulfilled it'll no longer show up within the output of kubectl get -n cert-manager challenges.

Also keep in mind that any DNS entries created to fulfill the challenge stage will typically have their TTL set to ~2min so if you go looking in your Cloudflare UI and do not see them, they likely already timed out and rolled off.

For e.g.:

References

@stevek-pro can you elaborate I'm not following the issue? Are we talking about this bit in the ClusterIssuer yaml? `namespace: cert-manager`? — slm, Mar 06 '22 at 03:18
@stevek-pro we talking about this bit? -- "If you want to create a single Issuer that can be consumed in multiple namespaces, you should consider creating a ClusterIssuer resource. This is almost identical to the Issuer resource, however is non-namespaced so it can be used to issue Certificates across all namespaces." — slm, Mar 06 '22 at 03:23

score 1 · Answer 3 · answered Jul 15 '22 at 10:54

I had this problem on DigitalOcean, for me disabling proxy protocol and tls-passthrough fixed the problem.

These configs should be commented on ingress-nginx service:

# Enable proxy protocol
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "true"
# Specify whether the DigitalOcean Load Balancer should pass encrypted data to backend droplets
service.beta.kubernetes.io/do-loadbalancer-tls-passthrough: "true"