Is there a way to force delete a crd with failing webhooks?

Question

The kubedb operator in this case has crashed and un-responsive - however I need to cleanup these resources.

 k delete redis r1 redis-queue --namespace cts --force --grace-period=0
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
Error from server (InternalError): Internal error occurred: failed calling webhook "redis.validators.kubedb.com": the server is currently unable to handle the request
Error from server (InternalError): Internal error occurred: failed calling webhook "redis.validators.kubedb.com": the server is currently unable to handle the request

Have you used the helm chart to deploy KubeDB? – Kamol Hasan Jun 16 '20 at 05:05 — Kamol Hasan, Jun 16 '20 at 05:05

score 18 · Answer 1 · edited Jan 07 '21 at 14:09

18

There should be a resource of kind MutatingWebhookConfiguration or ValidatingWebhookConfiguration which actually registers the webhook with Kubernetes API Server. You need to delete that resource first which deregisters the webhook from the Kubernetes API Server.

Temporary store kubedb-webhook configuration:

kubectl get ValidatingWebhookConfiguration redis.validators.kubedb.com \
  -o yaml > redis-validator.yaml
kubectl get MutatingWebhookConfiguration redis.validators.kubedb.com \
  -o yaml > redis-mutate-validator.yaml

Remove your resources:

kubectl delete redis r1 redis-queue --namespace cts

Bring back the webhook configuration:

kubectl apply -f redis-validator.yaml
kubectl apply -f redis-mutate-validator.yaml

edited Jan 07 '21 at 14:09

arve0

3,424
26
33

answered Jun 15 '20 at 16:10

Arghya Sadhu

41,002
9
78
107

unfortunately that didn't work ```k delete redis r1 redis-queue --namespace abc redis.kubedb.com "r1" deleted redis.kubedb.com "redis-queue" deleted`` I see this output but the terminal doesn't come back . – user2062360 Jun 15 '20 at 17:37
2

After deleting the webhook configuration, check that the custom resource does not have any finalizers set which will prevent deletion: `kubectl get redis -o custom-columns=name:.metadata.name,finalizers:.metadata.finalizers` and if so then edit the object and remove the finalizers from .metadata.finalizers and try to delete it again. – Jason Kincl Jul 06 '20 at 16:50

Is there a way to force delete a crd with failing webhooks?

1 Answers1