Failed calling webhook "namespace.sidecar-injector.istio.io"

Question

I have make my deployment work with istio ingressgateway before. I am not aware of any changes made in istio or k8s side.

When I tried to deploy, I see an error in replicaset side that's why it cannot create new pod.

Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc:443/inject?timeout=10s": dial tcp 10.104.136.116:443: connect: no route to host

When I try to go inside api-server and ping 10.104.136.116 (istiod service IP) it just hangs.

What I have tried so far:

Deleted all coredns pods
Deleted all istiod pods
Deleted all weave pods
Reinstalling istio via istioctl x uninstall --purge
turning all of VMs firewall
sudo iptables -P INPUT ACCEPT sudo iptables -P FORWARD ACCEPT sudo iptables -P OUTPUT ACCEPT sudo iptables -F
restarted all of the nodes
manual istio pod injection

Setup

k8s version: 1.21.2
istio: 1.10.3
HA setup
CNI: weave
CRI: containerd

Did you see [this question](https://stackoverflow.com/questions/60311316/istio-sidecar-injection-fails-with-no-route-to-host)? Do you have connection between your api-server and sidecar-injector? — Mikołaj Głodziak, Aug 02 '21 at 08:03
from inside kube-apiserver pod. tried to connect to 10.104.136.116:443 and it can't. but it can connect when I used the IPs of istiod(sidecar-injector) pod. — letthefireflieslive, Aug 02 '21 at 09:43
I don't have the visibility and control on the VM, lower network level. I am not sure if something have changed, I tested that this is working before. I already made this work by changing my CNI from weave to flannel. k8s version 1.21.3, containerd 1.3.7 — letthefireflieslive, Aug 02 '21 at 09:46

score 3 · Answer 1 · answered Apr 11 '22 at 08:32

3

In my case this was related to firewall. More info can be found here.

The gist of it is that on GKE at least you need to open another port 15017 in addition to 10250 and 443. This is to allow communication from your master node(s) to you VPC.

answered Apr 11 '22 at 08:32

Bart C

1,509
2
16
17

score 2 · Answer 2 · answered Aug 02 '21 at 09:49

2

I don't have a definite answer unto why is this happening. But kube-apiserver cannot access istiod via service IP, wherein it can connect when I used the istiod pod IP.

Since I don't have the control over the VM and lower networking layer and not sure if they have changed something (because it is working before).

I made this work by changing my CNI from weave to flannel

answered Aug 02 '21 at 09:49

letthefireflieslive

11,493
11
37
61

Well, I am having the same issue. https://stackoverflow.com/questions/69985819/kubernetes-deployments-are-failed-with-istio-sidecar-injection any suggestion? – Sachith Muhandiram Nov 16 '21 at 11:29
1

Hello, could you please try to go inside kube-api server and then try if you can connect to istiod via service IP and pod IP? if that's the case we have same issue – letthefireflieslive Nov 17 '21 at 03:18
@letthefirefileslive Well, `ping`,`telnet` and `curl` not available inside `api-server`. How did you check the connectivity? – Sachith Muhandiram Nov 17 '21 at 08:13

score 1 · Accepted Answer · answered Feb 15 '23 at 23:12

In my case it was due to firewall. Following this Istio debug guide, I identified that the kubectl get --raw /api/v1/namespaces/istio-system/services/https:istiod:https-webhook/proxy/inject -v4 command was timing out while all other cluster internal calls were ok.

The best way to diagnose this is to open temporarly your AWS Security Groups involved to 0.0.0.0/0 for port 15017 and then try again. If the errror won't show again, you know there's need to fix this part.

I am using EKS with Amazon VPC CNI v1.12.2-eksbuild.1

Failed calling webhook "namespace.sidecar-injector.istio.io"

3 Answers3