Kubernetes: No Route to Host

Question

I have a Bare-Metal Kubernetes custom setup (manually setup cluster using Kubernetes the Hard Way). Everything seems to work, but I cannot access services externally.

I can get the list of services when curl:

https://<ip-addr>/api/v1/namespaces/kube-system/services

However, when I try to proxy (using kubectl proxy, and also by using the <master-ip-address>:<port>):

https://<ip-addr>/api/v1/namespaces/kube-system/services/toned-gecko-grafana:80/proxy/

I get:

Error: 'dial tcp 10.44.0.16:3000: connect: no route to host'
Trying to reach: 'http://10.44.0.16:3000/'

• Even if I normally curl http://10.44.0.16:3000/ I get the same error. This is the result whether I curl from inside the VM where Kubernetes is installed. Was able to resolve this, check below.

• I can access my services externally using NodePort.

• I can access my services if I expose them through Nginx-Ingress.

• I am using Weave as CNI, and the logs were normal except a couple of log-lines at the beginning about it not being able to access Namespaces (RBAC error). Though logs were fine after that.

• Using CoreDNS, logs look normal. APIServer and Kubelet logs look normal. Kubernetes-Events look normal, too.

• Additional Note: The DNS Service-IP I assigned is 10.3.0.10, and the service IP range is: 10.3.0.0/24, and POD Network is 10.2.0.0/16. I am not sure what 10.44.x.x is or where is it coming from.

  • Also, I am using Nginx-Ingress (Helm Chart: https://github.com/helm/charts/tree/master/stable/nginx-ingress)

Here is output from one of the services:

{
  "kind": "Service",
  "apiVersion": "v1",
  "metadata": {
    "name": "kubernetes-dashboard",
    "namespace": "kube-system",
    "selfLink": "/api/v1/namespaces/kube-system/services/kubernetes-dashboard",
    "uid": "5c8bb34f-c6a2-11e8-84a7-00163cb4ceeb",
    "resourceVersion": "7054",
    "creationTimestamp": "2018-10-03T00:22:07Z",
    "labels": {
      "addonmanager.kubernetes.io/mode": "Reconcile",
      "k8s-app": "kubernetes-dashboard",
      "kubernetes.io/cluster-service": "true"
    },
    "annotations": {
      "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"v1\",\"kind\":\"Service\",\"metadata\":{\"annotations\":{},\"labels\":{\"addonmanager.kubernetes.io/mode\":\"Reconcile\",\"k8s-app\":\"kubernetes-dashboard\",\"kubernetes.io/cluster-service\":\"true\"},\"name\":\"kubernetes-dashboard\",\"namespace\":\"kube-system\"},\"spec\":{\"ports\":[{\"port\":443,\"targetPort\":8443}],\"selector\":{\"k8s-app\":\"kubernetes-dashboard\"}}}\n"
    }
  },
  "spec": {
    "ports": [
      {
        "protocol": "TCP",
        "port": 443,
        "targetPort": 8443,
        "nodePort": 30033
      }
    ],
    "selector": {
      "k8s-app": "kubernetes-dashboard"
    },
    "clusterIP": "10.3.0.30",
    "type": "NodePort",
    "sessionAffinity": "None",
    "externalTrafficPolicy": "Cluster"
  },
  "status": {
    "loadBalancer": {

    }
  }
}

I am not sure how to debug this, even some pointers to the right direction would help. If anything else is required, please let me know.

---

Output from kubectl get svc:

NAME                   TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
coredns-primary        ClusterIP   10.3.0.10    <none>        53/UDP,53/TCP,9153/TCP   4h51m
kubernetes-dashboard   NodePort    10.3.0.30    <none>        443:30033/TCP            4h51m

---

EDIT:

Turns out I didn't have kube-dns service running for some reason, despite having CoreDNS running. It was as mentioned here: https://github.com/kubernetes/kubeadm/issues/1056#issuecomment-413235119

Now I can curl from inside the VM successfully, but the proxy-access still gives me the same error: No route to host. I am not sure why or how would this fix the issue, since I don't see DNS being in play here, but it fixed the issue regardles. Would appreciate any possible explanation on this too.

Answer 1

I encountered the same issue and resolved it by running the commands below:

iptables --flush
iptables -tnat --flush
systemctl stop firewalld
systemctl disable firewalld
systemctl restart docker

Answer 2

for me the the solution was to modify the rules in iptables as described here

sudo iptables -D  INPUT -j REJECT --reject-with icmp-host-prohibited
sudo iptables -D  FORWARD -j REJECT --reject-with icmp-host-prohibited

Answer 3

When you are using kubectl proxy, by default you should use 127.0.0.1:8001 as HTTP Kube API URL. Your requests to http://127.0.0.1:8001 are then augmented with authentication headers and passed to API server. Thus you should try http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/toned-gecko-grafana:80/proxy/ rather then with https and api ip

Also, make sure you have socat installed on kube nodes.

Answer 4

Try to run curl https://<master-ip-address>:<port>.

If port is open you should receive a message related to certificate or HTTPS.

It port is closed (which is probably the issue in your case) - the no route to host message appears.

(!) Do not disable iptables.

Open the port by SSH-ing into master node and running:
A ) sudo firewall-cmd --add-port=<relevant-port>/tcp --permanent (I'm assuming firewalld is installed).

B ) sudo firewall-cmd --reload.

C ) sudo firewall-cmd --list-all - you should see <relevant-port> is updated.

public
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 
  services: dhcpv6-client ssh
  ports: <relevant-port>/tcp <---- Here
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: