Kubernetes outgoing connections

Question

I am going to keep this simple and ask, is there a way to see which pod have an active connection with an endpoint like a database endpoint?

My cluster contains a few hundred of namespace and my database provider just told me that the maximum amount of connections is almost reached and I want to pinpoint the pod(s) that uses multiple connections to our database endpoint at the same time.

I can see from my database cluster that the connections come from my cluster node's IP... but it won't say which pods... and I have quite lot of pods...

Thanks for the help

you need a centeral monitoring solution to monitor such connections and alot of other things going on , you should be able to see those things before you DB guy tells you that. — Ijaz Ahmad, Nov 20 '18 at 21:28
You are right, I do need more monitoring, but those connections are not exactly intuitive to monitor. — Sartigan, Nov 21 '18 at 15:00
Have you tried already the way of getting these statistics as suggested by Rico ? I tried to follow them exactly step by step and it didn`t work for me. — Nepomucen, Nov 21 '18 at 17:08
netstat didnt return any established connection to our database endpoint... so... it did not work either — Sartigan, Nov 21 '18 at 18:25

VAS · Answer 1 · 2019-01-24T16:06:21.443

Each container uses its own network namespace, so to check the network connection inside the container namespace you need to run command inside that namespace.

Luckily all containers in a Pod share the same network namespace, so you can add small sidecar container to the pod that print to the log open connections.

Alternatively, you can run netstat command inside the pod (if the pod has it on its filesystem):

kubectl get pods | grep Running | awk '{ print $1 }' | xargs -I % sh -c 'echo == Pod %; kubectl exec -ti % -- netstat -tunaple' >netstat.txt
# or
kubectl get pods | grep Running | awk '{ print $1 }' | xargs -I % sh -c 'echo == Pod %; kubectl exec -ti % -- netstat -tunaple | grep ESTABLISHED' >netstat.txt

After that you'll have a file on your disk (netstat.txt) with all information about connections in the pods.

The third way is most complex. You need to find the container ID using docker ps and run the following command to get PID

$ pid = "$(docker inspect -f '{{.State.Pid}}' "container_name | Uuid")"

Then, you need to create named namespace: (you can use any name you want, or container_name/Uuid/Pod_Name as a replacement to namespace_name)

sudo mkdir -p /var/run/netns
sudo ln -sf /proc/$pid/ns/net "/var/run/netns/namespace_name"

Now you can run commands in that namespace:

sudo ip netns exec "namespace_name" netstat -tunaple | grep ESTABLISHED

You need to do that for each pod on each node. So, it might be useful to troubleshoot particular containers, but it needs some more automation for your task.

It might be helpful for you to install Istio to your cluster. It has several interesting features mentioned in this answer

Here is another way to get all containers' connections on a node using docker: https://stackoverflow.com/questions/37171909/when-using-docker-established-connections-dont-appear-in-netstat — VAS, Mar 01 '19 at 17:09

score 0 · Answer 2 · answered Nov 20 '18 at 19:49

The easiest way is to run netstat on all your Kubernetes nodes:

$ netstat -tunaple | grep ESTABLISHED | grep <ip address of db provider>

The last column is the PID/Program name column, and that's a program that is running in a container (with a different internal container PID) in your pod on that specific node. There are all kinds of different ways to find out which container/pod it is. For example,

# Loop through all containers on the node with
$ docker top <container-id>

Then after you find the container id, if you look through all your pods:

$ kubectl get pod <pod-id> -o=yaml

And you can find the status, for example:

  status:
    conditions:
    - lastProbeTime: null
      lastTransitionTime: 2018-11-09T23:01:36Z
      status: "True"
      type: Initialized
    - lastProbeTime: null
      lastTransitionTime: 2018-11-09T23:01:38Z
      status: "True"
      type: Ready
    - lastProbeTime: null
      lastTransitionTime: 2018-11-09T23:01:38Z
      status: "True"
      type: ContainersReady
    - lastProbeTime: null
      lastTransitionTime: 2018-11-09T23:01:36Z
      status: "True"
      type: PodScheduled
    containerStatuses:
    - containerID: docker://f64425b3cd0da74a323440bcb03d8f2cd95d3d9b834f8ca5c43220eb5306005d

Kubernetes outgoing connections

2 Answers2