1

I am running K8S cluster on-premise (Nothing in the cloud) with one K8S Master and two worker nodes.

  • k8s-master : 192.168.100.100
  • worker-node-1 : 192.168.100.101
  • worker-node-2 : 192.168.100.102

I used kubernetes / ingress-nginx for routing traffic to my simple App. These are my pods running on both workers nodes:

[root@k8s-master ingress]# kubectl get pods -A -o wide
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE   IP                NODE            NOMINATED NODE   READINESS GATES
default                hello-685445b9db-b7nql                       1/1     Running   0          44m   10.5.2.7          worker-node-2   <none>           <none>
default                hello-685445b9db-ckndn                       1/1     Running   0          44m   10.5.2.6          worker-node-2   <none>           <none>
default                hello-685445b9db-vd6h2                       1/1     Running   0          44m   10.5.1.18         worker-node-1   <none>           <none>
default                ingress-nginx-controller-56c75d774d-p7whv    1/1     Running   1          30h   10.5.1.14         worker-node-1   <none>           <none>
kube-system            coredns-74ff55c5b-s8zss                      1/1     Running   12         16d   10.5.0.27         k8s-master      <none>           <none>
kube-system            coredns-74ff55c5b-w6rsh                      1/1     Running   12         16d   10.5.0.26         k8s-master      <none>           <none>
kube-system            etcd-k8s-master                              1/1     Running   12         16d   192.168.100.100   k8s-master      <none>           <none>
kube-system            kube-apiserver-k8s-master                    1/1     Running   12         16d   192.168.100.100   k8s-master      <none>           <none>
kube-system            kube-controller-manager-k8s-master           1/1     Running   14         16d   192.168.100.100   k8s-master      <none>           <none>
kube-system            kube-flannel-ds-76mt8                        1/1     Running   1          30h   192.168.100.102   worker-node-2   <none>           <none>
kube-system            kube-flannel-ds-bfnjw                        1/1     Running   10         16d   192.168.100.101   worker-node-1   <none>           <none>
kube-system            kube-flannel-ds-krgzg                        1/1     Running   13         16d   192.168.100.100   k8s-master      <none>           <none>
kube-system            kube-proxy-6bq6n                             1/1     Running   1          30h   192.168.100.102   worker-node-2   <none>           <none>
kube-system            kube-proxy-df8fn                             1/1     Running   13         16d   192.168.100.100   k8s-master      <none>           <none>
kube-system            kube-proxy-z8q2z                             1/1     Running   10         16d   192.168.100.101   worker-node-1   <none>           <none>
kube-system            kube-scheduler-k8s-master                    1/1     Running   12         16d   192.168.100.100   k8s-master      <none>           <none>
kubernetes-dashboard   dashboard-metrics-scraper-799cd98cf6-zh8xs   1/1     Running   9          16d   192.168.100.101   worker-node-1   <none>           <none>
kubernetes-dashboard   kubernetes-dashboard-74d688b6bc-hvxgm        1/1     Running   10         16d   10.5.1.17         worker-node-1   <none>           <none>

And these are the services running on my cluster:

[root@k8s-master ingress]# kubectl get svc
NAME                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
hello                                NodePort    10.105.236.241   <none>        80:31999/TCP                 30h
ingress-nginx-controller             NodePort    10.110.141.41    <none>        80:30428/TCP,443:32682/TCP   30h
ingress-nginx-controller-admission   ClusterIP   10.109.15.31     <none>        443/TCP                      30h
kubernetes                           ClusterIP   10.96.0.1        <none>        443/TCP                      16d

And this is the ingress description:

[root@k8s-master ingress]# kubectl describe  ingress ingress-hello
Name:             ingress-hello
Namespace:        default
Address:          10.110.141.41
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
Rules:
  Host        Path  Backends
  ----        ----  --------
  *
              /hello   hello:80 (10.5.1.18:80,10.5.2.6:80,10.5.2.7:80)
Annotations:  kubernetes.io/ingress.class: nginx
              nginx.ingress.kubernetes.io/rewrite-target: /

The issue is when accessing the first node by visiting worker-node-1 IP Address with Ingress Controller Port = 30428, http://192.168.100.101:30428, its working fine with no problems. While accessing worker-node-2 by visiting IP with same ingress port 30428, its NOT RESPONDING from out side the node and also from inside the node too by accessing the URL: http://192.168.100.102:30428 . I also tried executing telnet command (inside the worker node 2), no luck also:

[root@worker-node-2 ~]# telnet 192.168.100.102 30428
Trying 192.168.100.102...

The most interesting thing is the port is shows up in netstat command, as I am executing this command from inside the Node-2 , showing ingress Port:30428 is in LISTEN state:

[root@worker-node-2 ~]# netstat -tulnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      1284/kubelet
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      2578/kube-proxy
tcp        0      0 0.0.0.0:32682           0.0.0.0:*               LISTEN      2578/kube-proxy
tcp        0      0 0.0.0.0:111             0.0.0.0:*               LISTEN      1/systemd
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      1856/dnsmasq
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1020/sshd
tcp        0      0 127.0.0.1:631           0.0.0.0:*               LISTEN      1016/cupsd
tcp        0      0 127.0.0.1:41561         0.0.0.0:*               LISTEN      1284/kubelet
tcp        0      0 0.0.0.0:30428           0.0.0.0:*               LISTEN      2578/kube-proxy
tcp        0      0 0.0.0.0:31999           0.0.0.0:*               LISTEN      2578/kube-proxy
tcp6       0      0 :::10250                :::*                    LISTEN      1284/kubelet
tcp6       0      0 :::111                  :::*                    LISTEN      1/systemd
tcp6       0      0 :::10256                :::*                    LISTEN      2578/kube-proxy
tcp6       0      0 :::22                   :::*                    LISTEN      1020/sshd
tcp6       0      0 ::1:631                 :::*                    LISTEN      1016/cupsd
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           929/avahi-daemon: r
udp        0      0 0.0.0.0:44997           0.0.0.0:*                           929/avahi-daemon: r
udp        0      0 192.168.122.1:53        0.0.0.0:*                           1856/dnsmasq
udp        0      0 0.0.0.0:67              0.0.0.0:*                           1856/dnsmasq
udp        0      0 0.0.0.0:111             0.0.0.0:*                           1/systemd

based on my understanding , all worker node must expose NodePort for "ingress controller" port which=30428??

Edited: I found that "ingress-nginx-controller-56c75d774d-p7whv" is deployed only on node-1. Do I need to make sure that the ingress-nginx controller is running on all nodes? how to achieve that if this statement is true?

Faris Rjoub
  • 65
  • 8
  • What os your node runs on? Can you try to exec into pod that runs for example on `worker-node-1` and try to reach pod that runs on `worker-node-2`? – acid_fuji Jan 22 '21 at 11:30
  • I used "Centos 8" , 3 Hyper V virtual machines (1 master, 2 nodes). Yes, i tried to run exec from one pod (running on node1 and try to ping another pod which located on Node 2) it's NOT reachable. Then , i tired to ping another pod located on same node1, its fine , I reached it. – Faris Rjoub Jan 22 '21 at 13:28
  • 1
    This is what I suspected. This is all related centos8 switching from iptables to nftables. Switching your CNI to calico might be solution as they [added support](https://github.com/projectcalico/calico/issues/2322) for nft by setting `ELIX_IPTABLESBACKEND=NFT`. This will tell calico to use nftables backend – acid_fuji Jan 22 '21 at 14:05
  • 1
    Thanks for your reply, Yes, I am currently using flannel. So, I will switch to calico and try it out. – Faris Rjoub Jan 22 '21 at 14:13
  • In my comment above there is an `F` missing. It should be `FELIX_IPTABLESBACKEND=NFT`. – acid_fuji Jan 22 '21 at 14:22
  • 1
    Thanks @thomas for your comments, I managed to switch CNI to calico with setting the new env you mentioned, all nodes are now talks to each others smoothly. – Faris Rjoub Jan 23 '21 at 12:44
  • Great news! I'll place an answer then :) – acid_fuji Jan 23 '21 at 18:17

1 Answers1

2

Kubernetes networking (kube-proxy to be more specific) uses iptables to control the network connections between pods and nodes. Since Centos 8 uses nftables instead iptables this cause networking issues.

Calico in v.3.8.1+ included support for hosts which uses iptables in NFT mode. The solution is to set FELIX_IPTABLESBACKEND=NFT option. This will tell Calico to use nftables backed.

This parameter controls which variant of iptables binary Felix uses. Set this to Auto for auto detection of the backend. If a specific backend is needed then use NFT for hosts using a netfilter backend or Legacy for others. [Default: Legacy]

Please visit this calico page to check how to configure felix. For more reading please visit this github issues.

acid_fuji
  • 6,287
  • 7
  • 22