Why my ClusterIP does not work with the ip assigned?

Question

I have defined a new service with a ClusterIP.

[ciuffoly@master-node ~]$ kubectl get services
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
kubernetes   ClusterIP      10.96.0.1       <none>          443/TCP          4d1h    
test-reg     ClusterIP      10.102.196.35   <none>          5000/TCP         58m
test-web     LoadBalancer   10.108.151.13   192.168.1.125   80:30001/TCP     73m

The pod is running on worker-node1 and I can connect to this server with the worker-node1 plumbed on ethernet ip.

[ciuffoly@worker-node1 ~]$ ip addr show|grep "192\.168\.1\."
inet 192.168.1.20/24 brd 192.168.1.255 scope global noprefixroute ens33

[ciuffoly@worker-node1 ~]$ telnet 192.168.1.20 5000    
Connected to 192.168.1.20.
Escape character is '^]'.
^]
telnet> q

[ciuffoly@master-node ~]$ telnet 192.168.1.20 5000
Connected to 192.168.1.20.
Escape character is '^]'.
^]
telnet> q

But I cannot connect to this service with the ClusterIP

[ciuffoly@master-node ~]$ telnet 10.102.196.35 5000
Trying 10.102.196.35...
^C

Following the answers I have tested also NodePort but I still have the same problem.

[ciuffoly@master-node ~]$ kubectl get services|grep reg
test-reg     NodePort       10.111.117.116   <none>          5000:30030/TCP   5m41s

[ciuffoly@master-node ~]$ kubectl delete svc test-reg
service "test-reg" deleted
[ciuffoly@master-node ~]$ netstat -an|grep 30030

[ciuffoly@master-node ~]$ kubectl apply -f myreg.yaml
myreg.yamldeployment.apps/test-reg unchanged
service/test-reg created
[ciuffoly@master-node ~]$ netstat -an|grep 30030
tcp        0      0 0.0.0.0:30030           0.0.0.0:*               LISTEN

This does not work

[ciuffoly@master-node ~]$ telnet master-node  30030
Trying 192.168.1.10...
^C

This works

[ciuffoly@master-node ~]$ telnet worker-node1  30030
Trying 192.168.1.20...
Connected to worker-node1.
Escape character is '^]'.
^]
telnet> q
Connection closed.

Answer 1

This is a community wiki answer posted for better visibility. Feel free to expand it.

As already confirmed by andrea ciuffoli, switching from Flannel to Calico solved the issue.

Flannel is a very simple overlay network that satisfies the Kubernetes requirements.

On the other hand, Calico is an open source networking and network security solution for containers, virtual machines, and native host-based workloads. Calico supports multiple data planes including: a pure Linux eBPF dataplane, a standard Linux networking dataplane, and a Windows HNS dataplane. Calico provides a full networking stack but can also be used in conjunction with cloud provider CNIs to provide network policy enforcement.

It's hard to say what was the sole reason behind the final solution but you can find some details about Comparing Kubernetes CNI Providers: Flannel, Calico, Canal, and Weave.

Answer 2

From the docs:

The service type ClusterIP exposes the Service on a cluster-internal IP. Choosing this value makes the Service only reachable from within the cluster. This is the default ServiceType.

So you can reach your ClusterIP service only from within your cluster. So you could deploy a pod with telnet installed and test your setup from there.

If you want to connect from your host, you could use the service type NodePort.

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: NodePort
  selector:
    app: MyApp
  ports:
    - port: 80
      targetPort: 80
      nodePort: 30080

This way you could connect to the service via hostname:30080. The thing is, that you can use every hostname of your cluster.