0

I have an EKS Kubernetes cluster. High level the setup is:

a) There is an EC2 instance, lets call it "VM" or "Host"

b) In the VM, there is a POD running 2 containers: Side Car HAProxy Container + MyApp Container

What happens is that when external requests come, inside of HAProxy container, I can see that the source IP is the "Host" IP. As the Host has a single IP, there can be a maximum of 64K connections to HAProxy.

I'm curious to know how to workaround this problem as I want to be able to make like 256K connections per Host.

David Buck
  • 3,752
  • 35
  • 31
  • 35
Luis Serrano
  • 29
  • 11
  • 64k connections in same time? Are you sure that bottle neck is number of connections? Even if data transfer for one connection will be 1 Byte/s you must handle 64MB/s in total by one EC2 instance. And... is one instance may be called `high availability` proxy? – rzlvmp Feb 04 '22 at 04:54
  • [275k requests per second](https://threatpost.com/javascript-ddos-attack-peaks-at-275000-requests-per-second/114828/#:~:text=JavaScript%2DBased%20DDoS%20Peaks%20at%20275%2C000%20Requests%20Per%20Second%20%7C%20Threatpost) is a massive DDOS attack for whole service. And you plan to handle all of it by one "Host"? – rzlvmp Feb 04 '22 at 05:03
  • These are connections which are very quiet in general and send very few data. These are not requests per second. My limitation is the single source Ip in the haproxy side (which is the IP of the Host), that limits to 64k the max number of connections – Luis Serrano Feb 04 '22 at 05:13
  • I added explanation about `64k` meaning and how it works and what to do in theory. However, if your question `how to workaround this problem` means `how exactly configure EKS cluster` then that will be useless, just ignore it – rzlvmp Feb 04 '22 at 06:30

2 Answers2

1

I'm not sure is you understand reason for 64k limit so try to explain it

At first that is a good answer about 64k limitations

Let's say that HAProxy (192.168.100.100) listening at port 8080 and free ports at Host (192.168.1.1) are 1,353~65,353, so you have combination of:

source 192.168.1.1:1353~65353 → destination 192.168.100.100:8080

That is 64k simultaneous connections. I don't know how often NAT table is updating, but after update unused ports will be reused. So simultaneous is important

If your only problem is limit of connections per IP, here is couple solutions:

  1. Run multiple HAProxyes. Three containers increase limit to 64,000 X 3 = 192,000
  2. Listen multiple ports on HAProxy (check about SO_REUSEPORT). Three ports (8080, 8081, 8082) increase max number of connections to 192,000

Host interface IP is acting like a gateway for Docker internal network so I not sure if it is possible to set couple IPs for Host or HAProxy. At least I didn't find information about it.

rzlvmp
  • 7,512
  • 5
  • 16
  • 45
0

It turns that in Kubernetes one can configure how we want clients to access the service and the choice that we had was nodePort. When we changed it to hostPort, the source IP was seen in the haproxy container and hence the limitation that I was having was removed.

If this option would have failed, my next option was to try the recommendation in the other response which was to have haproxy listening in multiple ports. Thankfully that was not needed.

Thanks!

Luis Serrano
  • 29
  • 11