netstat showing foreign ports as kubernetes:port. What does this mean?

Question

I am using a Windows 10 Pro machine.

When I run netstat, it is showing kubernetes:port as foreign address in active connections.

What does this mean? I have checked and there is no kubernetes cluster running on my machine.

How do I close these connections?

Minikube status:

$ minikube status
host:
kubelet:
apiserver:
kubectl:

may I know why you added `minikube` tag and say there is no kubernetes cluster on your machine? — Vit, Jan 27 '20 at 16:28
I installed minikube to run the kubernetes, then i stopped the minikube and k8s cluster with it. That's why i mentioned. — Shubham Chadokar, Jan 27 '20 at 17:32

score 41 · Accepted Answer · edited Jun 04 '20 at 00:48

That happens because of the way netstat renders output. It has nothing to do with actual Kubernetes.

I have Docker Desktop for Windows and it adds this to the hosts file:

# Added by Docker Desktop
192.168.43.196 host.docker.internal
192.168.43.196 gateway.docker.internal
# To allow the same kube context to work on the host and the container:
127.0.0.1 kubernetes.docker.internal
# End of section

There is a record which maps 127.0.0.1 to kubernetes.docker.internal. When netstat renders its output, it resolves foreign address and it looks at the hosts file and sees this record. It says kubernetes and that is what you see in the console. You can try to change it to

127.0.0.1 tomato.docker.internal

With this, netstat will print:

  Proto  Local Address          Foreign Address        State
  TCP    127.0.0.1:6940         tomato:6941            ESTABLISHED
  TCP    127.0.0.1:6941         tomato:6940            ESTABLISHED
  TCP    127.0.0.1:8080         tomato:40347           ESTABLISHED
  TCP    127.0.0.1:8080         tomato:40348           ESTABLISHED
  TCP    127.0.0.1:8080         tomato:40349           ESTABLISHED

So what actually happens is there are connections from localhost to localhost (netstat -b will show apps that create them). Nothing to do with Kubernetes.

Thanks for the detailed explanation. I checked the host file and removed kubernetes reference and now there is no kube as foreign host. — Shubham Chadokar, Jan 05 '21 at 17:51
`netstat -b` saved the day for me. Turned out to be `openvpn.exe` established a connection called `kubernetes:1099` which messes up a default JMX manager — rupweb, Sep 30 '21 at 12:02

score 8 · Answer 2 · edited Jun 04 '20 at 00:49

8

It seems that Windows docker changed your hosts file. So, if you want to get rid of these connections, just comment out the corresponding lines in the hosts file.

The hosts file on Windows 10 is located in C:\Windows\System32\drivers\etc and the records may look something like 127.0.0.1 kubernetes.docker.internal. I am pretty sure it will disrupt your docker service on Windows (yet, I am not an expert), so don't forget to uncomment these lines whenever you need to get the docker service back.

edited Jun 04 '20 at 00:49

Pang

9,564
146
81
122

answered Mar 03 '20 at 16:34

Leo Skhrnkv

1,513
16
27

it doesn't get rid of the instances or/and ports. They call themselves cbs:XXXXXX now. I assume they came bundled with docker. It's strange, that when firefox get's opened, more of these instances appear. Something fishy is going on here. – Nils Riga May 24 '21 at 12:36
I didn't want to remove the docker additions from the hosts file, so I just added a line at the top of the hosts file for `127.0.0.1 localhost` and then connections to localhost were correctly identified by the system name again. – Travis Britz Mar 24 '22 at 03:46

score 1 · Answer 3 · edited Jun 20 '20 at 09:12

OK, it looks like your minikube instance is definitely deleted. Keep in mind that in Linux or other nix-based systems it is totally normal that many processes use network sockets to communicate between each other e.g. you will see many established connections with both local and foreign addresses set to localhost:

tcp        0      0 localhost:45402         localhost:2379          ESTABLISHED
tcp        0      0 localhost:45324         localhost:2379          ESTABLISHED
tcp        0      0 localhost:2379          localhost:45300         ESTABLISHED
tcp        0      0 localhost:45414         localhost:2379          ESTABLISHED
tcp        0      0 localhost:2379          localhost:45388         ESTABLISHED
tcp        0      0 localhost:40600         localhost:8443          ESTABLISHED

kubernetes in your case is nothing more than hostname of one of your machines/VMs/instances. Maybe the one on top of which you ran your minikube you called kubernetes and that's why this hostname appears currently in your active network connections. Basically it has nothing to do with running kubernetes cluster.

To make it clearer you may cat the content of your /etc/hosts file and look for the entry kubernetes. Then you can compare them with your network interfaces addresses (run ip -4 a). Most probably kubernetes entry in /etc/hosts is mapped to one of them.

Let me know if it clarifies your doubts.

EDIT:

I've reproduced it on Minikube on my linux instance and noticed exactly the same behaviour, but it looks like the ESTABLISHED connections are showing only after successfull minikube stop. After minikube delete they're gone. It looks like those connections indeed belong to various components of kubernetes, but for some reason are not terminated. Basically closing established network connections is responsibility of the application which creates them and it looks like for some reason minikube is not terminating them.

If you run:

sudo netstat -ntp ### important: it must be run as superuser

it shows additionally PID/Program name column in which you can see by which program specific connection was established. You will see a lot of ESTABLISHED network connections belonging to etcd and kube-apiserver.

First I try to reboot the whole instance. It obviously close all the connections but then I verified a few times and it looks like successfully performed minikube delete also closes all connections.

Additionally you may want to check available docker containers by running:

docker ps

or:

docker container ls

After stopping the minikube instance it still shows those containers and it looks like the reason why a lot of connections with certain kubernetes components** are still shown by netstat command.

However after minikube delete neither containers nor ESTABLISHED connections with kubernetes cluster components are available any more.

Yes, I know that, but if you are using `minikube` or `docker desktop` it basically uses some kind of virtualization and runs your docker containers on top of linux os. — mario, Feb 11 '20 at 13:15
OK, so you run `netstat` in your `windows command line` or `powershell`, right ? — mario, Feb 11 '20 at 13:16

netstat showing foreign ports as kubernetes:port. What does this mean?

3 Answers3

EDIT:

Linked