88

We created a kubernetes cluster for a customer about one year ago with two environments; staging and production separated in namespaces. We are currently developing the next version of the application and need an environment for this development work, so we've created a beta environment in its own namespace.

This is a bare metal kubernetes cluster with MetalLB and and nginx-ingress. The nginx ingress controllers is installed with helm and the ingresses are created with the following manifest (namespaces are enforced by our deployment pipeline and are not visible in the manifest):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: api-ingress
  annotations:
    #ingress.kubernetes.io/ssl-redirect: "true"
    #kubernetes.io/tls-acme: "true"
    #certmanager.k8s.io/issuer: "letsencrypt-staging"
    #certmanager.k8s.io/acme-challenge-type: http01
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers "X-Robots-Tag: noindex, nofollow";
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, OPTIONS"
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
spec:
  tls:
    - hosts:
        - ${API_DOMAIN}
      secretName: api-cert
  rules:
    - host: ${API_DOMAIN}
      http:
        paths:
          - backend:
              serviceName: api
              servicePort: 80

When applying the manifest kubernetes responds with the following error:

Error from server (InternalError): error when creating "STDIN": Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: service "ingress-nginx-controller-admission" not found

I've attempted to update the apiVersion of the ingress manifest to networking.k8s.io/v1beta1 (this is the apiVersion the new nginx-ingress controllers are installed with via helm), but I'm getting the same error.

My initial suspicion is that this is related to a change in the nginx-ingress between the current installation and the installation from one year ago, even if the ingress controllers are separated by namespaces. But i cant find any services called ingress-nginx-controller-admission in any of my namespaces, so I'm clueless how to proceed.

Jørgen
  • 3,467
  • 6
  • 33
  • 49

6 Answers6

260

I had the same problem and found a solution from another SO thread.

I had previously installed nginx-ingress using the manifests. I deleted the namespace it created, and the clusterrole and clusterrolebinding as noted in the documentation, but that does not remove the ValidatingWebhookConfiguration that is installed in the manifests, but NOT when using helm by default. As Arghya noted above, it can be enabled using a helm parameter.

Once I deleted the ValidatingWebhookConfiguration, my helm installation went flawlessly.

kubectl delete -A ValidatingWebhookConfiguration ingress-nginx-admission
Patrick Gardella
  • 3,971
  • 1
  • 17
  • 17
  • Mine didn't, it just hangs there forever. – Nathan McKaskle Jul 28 '21 at 18:17
  • on newer k8s versions its 'kubectl delete -A validatingwebhookconfigurations.admissionregistration.k8s.io ....' – lexXxel Oct 19 '21 at 13:01
  • 1
    Thanks a ton, after deleting the already existing ValidatingWebhookConfiguration from previous install, the new deployment went smooth :) – Pinak Mazumdar Nov 18 '21 at 12:47
  • If the answer does not solve your problem permanently, try deleting *ingress-nginx-admission-create*, *ingress-nginx-admission-create* jobs with it. For delete jobs run this command ```kubectl delete job -n ingress-nginx ingress-nginx-admission-create ingress-nginx-admission-patch``` – Yeahia Md Abid Mar 13 '23 at 11:39
  • This should be in the faq on nginx-ingress for full cleanup. – Evan Carroll May 05 '23 at 18:18
30

You can check if there is a validation webhook and a service. If they don't exist double check the deployment and add these.

kubectl get -A ValidatingWebhookConfiguration
NAME                      CREATED AT
ingress-nginx-admission   2020-04-22T15:01:33Z

kubectl get svc -n ingress-nginx
NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                      AGE
ingress-nginx-controller             NodePort    10.96.212.217   <none>        80:32268/TCP,443:32683/TCP   2m34s
ingress-nginx-controller-admission   ClusterIP   10.96.151.42    <none>        443/TCP                      2m34s

Deployment yamls here have the webhook and service.

Since you have used helm to install it you can enable/disable the webhook via a helm parameter as defined here

Arghya Sadhu
  • 41,002
  • 9
  • 78
  • 107
  • Hi, thank you for responding. I verified that the webhook exists. The service ingress-nginx-controller exists, but in the beta namespace - not in the namespace ingress-nginx. I created the ingress-nginx-controller-admission service in the beta namespace, but I get the same error. – Jørgen Apr 23 '20 at 05:57
  • how many nginx ingress controller deployments you have in the same cluster? – Arghya Sadhu Apr 23 '20 at 05:59
  • Three. One for each environment – Jørgen Apr 23 '20 at 06:00
  • I hope you have followed this https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/#multiple-ingress-nginx-controllers. otherwise it will lead to issues – Arghya Sadhu Apr 23 '20 at 06:02
  • No, I've seen it now while troubleshooting this issue. I installed the two first environments at the same time using helm and they have both worked flawlessly for a year now. Could this be related? I might need to update the args for all environments then – Jørgen Apr 23 '20 at 06:07
  • You should maybe consider configuring all namespaces under a single ingress group, just add security and forwarding rules by configuring the nginx ingress. – James Stone Apr 24 '20 at 10:27
5

There is some issue with SSL cert it seems in the webhook.

Chaning failurePolicy: Fail to Ignore worked for me in the

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-0.32.0/deploy/static/provider/baremetal/deploy.yaml

for more info check:

https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/

Pankaj Gupta
  • 59
  • 1
1

my problem is proven to be a ssl cert issue. after I delete"ValidatingWebhookConfiguration", the issue is resolved

vincent Chen
  • 11
  • 1
1

For me issue was with Kubernetes version 1.18 and I upgraded to 1.19.1 and it worked just fine.

Pod status

k get pods -n ingress-nginx
NAME                                        READY   STATUS             RESTARTS   AGE
ingress-nginx-admission-create-cgpj7        0/1     Completed          0          3m44s
ingress-nginx-admission-patch-mksxs         0/1     Completed          0          3m44s
ingress-nginx-controller-5fb6f67b9c-ps67k   0/1     CrashLoopBackOff   5          3m45s

Error logs from pod

I0916 07:15:34.317477       8 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
F0916 07:15:34.318721       8 main.go:107] ingress-nginx requires Kubernetes v1.19.0 or higher

k get po -n ingress-nginx
NAME                                        READY   STATUS      RESTARTS   AGE
ingress-nginx-admission-create-2tk8p        0/1     Completed   0          104s
ingress-nginx-admission-patch-nlv5w         0/1     Completed   0          104s
ingress-nginx-controller-79c4d49bb9-7bgcj   1/1     Running     0          105s
Smit Jain
  • 141
  • 1
  • 7
0

I faced this issue when working on a Kubernetes cluster.

The issue arose when I was migrating resources from one nodepool to another nodepool in a test Kubernetes Cluster.

I forgot that I had not migrated out the Nginx ingress and the Cert Manager out of the noodpool that I wanted to decommission. So after migrating other applications out of the noodpool that I wanted to decommission I deleted the noodpool, which consequently deleted Nginx ingress and the Cert Manager from the Kubernetes Cluster.

All I had to do was to redeploy the Nginx ingress and the Cert Manager to the new noodpool.

Promise Preston
  • 24,334
  • 12
  • 145
  • 143