Kubernetes deployments are failed with istio-sidecar injection

Question

Our K8 cluster was working for more than a year, recently it got some strange behavior and now when we deploy an app using kubectl apply -f deployment-manifest.yaml, it doesnt show in kubectl get pods. But shows in kubectl get deployments with 0/3 state. kubectl describe deployment app-deployment

Conditions:
  Type             Status  Reason
  ----             ------  ------
  Available        False   MinimumReplicasUnavailable
  ReplicaFailure   True    FailedCreate
  Progressing      False   ProgressDeadlineExceeded

When I check kube-apiserver logs

I1115 12:55:56.110277       1 trace.go:116] Trace[16922026]: "Call validating webhook" configuration:istiod-istio-system,webhook:validation.istio.io,resource:networking.istio.io/v1alpha3, Resource=gateways,subresource:,operation:CREATE,UID:00c425da-6475-4ed3-bc25-5a81d866baf2 (started: 2021-11-15 12:55:26.109897413 +0000 UTC m=+8229.935658158) (total time: 30.00030708s):
Trace[16922026]: [30.00030708s] [30.00030708s] END
W1115 12:55:56.110327       1 dispatcher.go:128] Failed calling webhook, failing open validation.istio.io: failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: dial tcp 10.233.30.109:443: i/o timeout
E1115 12:55:56.110363       1 dispatcher.go:129] failed calling webhook "validation.istio.io": Post https://istiod.istio-system.svc:443/validate?timeout=30s: dial tcp 10.233.30.109:443: i/o timeout
I1115 12:55:56.121271       1 trace.go:116] Trace[576910507]: "Create" url:/apis/networking.istio.io/v1alpha3/namespaces/istio-system/gateways,user-agent:pilot-discovery/v0.0.0 (linux/amd64) kubernetes/$Format,client:192.168.1.16 (started: 2021-11-15 12:55:26.108861126 +0000 UTC m=+8229.934621868) (total time: 30.012357263s):

Kube-controller logs

I1116 07:55:06.218995       1 event.go:278] Event(v1.ObjectReference{Kind:"ReplicaSet", Namespace:"default", Name:"ops-executor-app-6647b7cbdb", UID:"0ef5fefd-88d7-480f-8a5d-f7e2c8025ae9", APIVersion:"apps/v1", ResourceVersion:"122334057", FieldPath:""}): type: 'Warning' reason: 'FailedCreate' Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istiod.istio-system.svc:443/inject?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E1116 07:56:12.342407       1 replica_set.go:535] sync "default/app-6769f4cb97" failed with Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istiod.istio-system.svc:443/inject?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

When I check kubectl get pods -n istio-system

NAME                                    READY   STATUS    RESTARTS   AGE
istio-egressgateway-794d6f956b-8p5vz    0/1     Running   5          401d
istio-ingressgateway-784f857457-2fz4v   0/1     Running   5          401d
istiod-67c86464b4-vjp4j                 1/1     Running   5          401d

egress and ingress gateway logs have

2021-11-15T16:55:31.419880Z error   citadelclient   Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 169.254.25.10:53: no such host"
2021-11-15T16:55:31.419912Z error   cache   resource:default request:37d26b55-df29-465f-9069-9b9a1904e8ab CSR retrial timed out: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 169.254.25.10:53: no such host"
2021-11-15T16:55:31.419956Z error   cache   resource:default failed to generate secret for proxy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 169.254.25.10:53: no such host"
2021-11-15T16:55:31.419981Z error   sds resource:default Close connection. Failed to get secret for proxy "router~10.233.70.87~istio-egressgateway-794d6f956b-8p5vz.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 169.254.25.10:53: no such host"
2021-11-15T16:55:31.420070Z info    sds resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2021-11-15T16:55:31.420336Z warning envoy config    StreamSecrets gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: lookup istiod.istio-system.svc on 169.254.25.10:53: no such host"
2021-11-15T16:55:48.020242Z warning envoy config    StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2021-11-15T16:55:48.020479Z warning envoy config    Unable to establish new stream
2021-11-15T16:55:51.025327Z info    sds resource:default new connection
2021-11-15T16:55:51.025597Z info    sds Skipping waiting for gateway secret

Tried to get details as described here, but it shows no resources.
Tried deploying application in non-istio injected namespace and it works without any issue.

We have baremetal cluster running Ubuntu-18.04LTS.

istioctl version

client version: 1.7.0
control plane version: 1.7.0
data plane version: none

Kubernetes v1.18.8

As described here, ran kubectl get --raw /api/v1/namespaces/istio-system/services/https:istiod:https-webhook/proxy/inject -v4

I1116 17:05:32.703339   28777 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "the server rejected our request for an unknown reason",
  "reason": "BadRequest",
  "details": {
    "causes": [
      {
        "reason": "UnexpectedServerResponse",
        "message": "no body found"
      }
    ]
  },
  "code": 400
}]
F1116 17:05:32.703515   28777 helpers.go:115] Error from server (BadRequest): the server rejected our request for an unknown reason

From ingres-gateway

istio-proxy@istio-ingressgateway-784f857457-2fz4v:/$ curl https://istiod.istio-system:443/inject -k

curl: (6) Could not resolve host: istiod.istio-system

Edit : in master node /var/lib/kubelet/config.yaml

clusterDNS:
- 169.254.25.10

and we can ping to this IP from our nodes.

I found this in coredns pod logs

E1123 08:57:05.386992       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.233.0.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused
E1123 08:57:05.387108       1 reflector.go:153] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Failed to list *v1.Service: Get https://10.233.0.1:443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.233.0.1:443: connect: connection refused

Do you know, you are using deprecated version of istio and Kubernetes? Could you update both to newest versions? Further, you have this message: `istiod.istio-system.svc on 169.254.25.10:53: no such host`. Could you check this host? — Mikołaj Głodziak, Nov 16 '21 at 12:41
@MikołajGłodziak Well, we can not update this as its in client's UAT. Yes, there are no such host in the system. — Sachith Muhandiram, Nov 16 '21 at 13:37
Some of the errors suggest the potential of trouble with kube-dns in kube-system namespace. Check that out, potentially. restart the deployment. — akokskis, Nov 17 '21 at 02:49
@akokskis `coredns` and `dns-autoscler` both are working. No errors there — Sachith Muhandiram, Nov 17 '21 at 08:07
Did you check your DNS settings? This error could be connected to problem with DNS. Another idea is expired SSL certificates. Could you check it? See also [this question](https://stackoverflow.com/questions/24967855/curl-6-could-not-resolve-host-google-com-name-or-service-not-known). — Mikołaj Głodziak, Nov 17 '21 at 13:06
@MikołajGłodziak You mean kube-dns? our kubernetes certificates were recently updated. — Sachith Muhandiram, Nov 17 '21 at 14:32
How is look like your `/etc/resolv.conf` file look like? Could you add it to the question? — Mikołaj Głodziak, Nov 19 '21 at 16:49
@MikołajGłodziak in which node? k8 master? or kubernetes pods? — Sachith Muhandiram, Nov 20 '21 at 02:17
On master node, could you change it to other value and test again? It seems like you have problem with resolve DNS for `istiod.istio-system.svc`. Did you try to connect to this host directly? — Mikołaj Głodziak, Nov 23 '21 at 07:17
I tried deploying application in non-istio enabled ns.. It also have the connectivity issue. — Sachith Muhandiram, Nov 23 '21 at 08:26
What exactly did this error look like? Do you have some logs? — Mikołaj Głodziak, Nov 25 '21 at 12:27
@MikołajGłodziak No deployment issues, only can not connect to the service, even telnet — Sachith Muhandiram, Nov 25 '21 at 12:34
Yes I understand, but what error you got when did you try to connect to this service (even telnet)? — Mikołaj Głodziak, Nov 26 '21 at 13:56
If your issue has been resolved please post the answer so, that community members can take it as a reference. — Abhijith Chitrapu, Sep 28 '22 at 10:31
@AbhijithChitrapu We had to re install k8 again to solve this issue. — Sachith Muhandiram, Sep 28 '22 at 11:13

Kubernetes deployments are failed with istio-sidecar injection

0 Answers0

Linked