How to create a software load balancer for socket connections in java

Question

I am planning to create a sofware load balancer that will sit in front of many socket servers in linux. The clients will connect to the load balancer . The load balancer will maintain a table of alternative ips and their ports. It will connect the client to the best available ip and port and disconnect itself from the client.Thus it will go out of the scene(will no longer be connected to the client).In this condition only connected devices will be the client and the new socket server but NOT the load balancer.

Example : Cleint ip 10.1.2.3 port 1234  
           load balancer Ip 10.1.2.4 port 1235
           list of socket servers   in Load Balancer: 
           A Ip 10.1.2.4 port 1236 
           B Ip 10.1.2.4 port 1237
           C  Ip 10.1.2.5 port 1238
    Now 
for the 1st request to the load balancer from client, the load balancer  will establish a connection between the client &  server A and disconnect itself from client.
     for the 2nd request to the load balancer from client, the load balancer  will establish a connection between the client &  server B and disconnect itself from client.
     for the 3rd request to the load balancer from client, the load balancer  will establish a connection between the client &  server C and disconnect itself from client.

Any Help on implementing this in Java is greatly appreciated.

Check *[this](http://stackoverflow.com/questions/3954454/fast-implementation-of-a-port-forward-in-java)* post — Extreme Coders, Apr 22 '13 at 07:58

score 2 · Answer 1 · answered Sep 24 '15 at 00:07

This miniature design can be helpful for apps in small devices like mobiles and tabs. There can be a limit set by the server on how many round trip can a request be allowed from a particular device. Off course scaling of the online servers will help reduce the round trip counts .

mantiq · Answer 2 · 2016-01-06T12:16:19.027

I would use redis to store the lookup table. Each load-balancer server will lookup in the lookup table in redis for a connection to the most available / most prioritized server . This lookup will return a single integer which is an index of the server. Each app in the client will store the server ip with their respective indexes. so this lookup is very fast and less than 30 ms. At this point the connection will be faster.NO redirect is needed. A fall back is also provided in case there is concurrent connections and the qouta on the desired server is finished by the time the app tries to connect to the looked up server. In this case it will again look up the most available server , ie, start over recursively until it gets connected successfully or all resourced are finished and the connection-request is marked as dead end.

How about reserving the connection for few milliseconds, for each lookup ? After the expiry of the delay to connect for that lookup, the page-file will be made available to occupy for a new connection. This will decrease the recursive lookup but also block the connectivity . The delay should be sufficient for a connection to establish which may vary. On the other hand new connections will be blocked during this delay , which can result in a bad user experience. You need to trade-off between these two: decrease lookup and block connectivity vs never block connectivity and endure recursive lookup which is very fast.

user2782766 · Answer 3 · 2013-10-06T12:20:19.143

1

The difficulty is disconnecting the device from server side and connecting it with the intended server.
There can be a work around : use redirect instead.

Each server is a load balancer and service provider to the client devices
Each server will always keep track of its open file limits and preserver safe margin from the max openfile limits
This pool will be used to check if any redirect is needed.
When the open file limit reaches the safe limit, any further connection request will return the next available server ip to the client device.
The next nearest available server ip can be maintained in a lookup table in memory.
The device will check if the returned value starts with the redirect ip it will re connect automatically to the received ip.Otherwise it will assume that the device got the service from any of the server successfully.

Thus we can avoid the open file limit and connection refused error .

edited Oct 06 '13 at 12:20

answered Oct 06 '13 at 12:03

user2782766

50
4

2

This will provide the isolated connection to the next available server. But cannot guarantee that the connection will succeed on second attempt. It may require a lot of round trip from server to the device. and every new connection is expensive. – App Work Oct 06 '13 at 12:41
If you can device a rule to the target redirect-server that has the highest availability in therms of number of opened files descriptors , then you can ensure the redirect on second attempt every time. The load balancer will read the open File descriptor value for each node , and return the best available server ip. – user2782766 Oct 08 '13 at 07:04

score 0 · Answer 4 · answered Apr 22 '13 at 09:26

0

I do not quite understand the requirement that the load balancer should disconnect from the client. If your sockets are in fact TCP connections, as they appear to be, I do not see how you could offload the connection to a client running somewhere else without some low level hackery. For example, look at ldirectord from linux virtual server. This allows fully offloading the connection.

For pure simplicity, I'd just use HAProxy. Does most of what you want, except for offloading the connection.

Finally, you could also use some kind of round-robin DNS solution. That would also offload the connection as you require.

answered Apr 22 '13 at 09:26

izak

981
7
10

Actually offloading of the load-balancer server is inevitable and should do as fast as possible (so that it can accept other connection requests). And the load-balaner should be as simple as possible just to establish the 3rd party connection to the client and offload itself. Thanks – App Work Apr 22 '13 at 10:33
But that is what puzzles me. In a proper multi-threaded load balancer multiple requests can be handled concurrently, so your load balancer does not become unavailable because it already handles another request. Usually it does have configurable limits though, and any connections beyond the limit will be queued. Here is a good [example](http://mike-k-linux-mumbling.blogspot.com/2012/01/postgresql-and-haproxy-queuing.html). – izak Apr 22 '13 at 12:45
Wait, I think I understand where you are heading. You need a java load balancer that takes an incoming connection and then starts a handler thread to handle that connection. The thread will live as long as the request takes to complete, but the parent process returns to accepting the next connection. Your requirement was specifically that this must be in java, which unfortunately means I cannot deliver. Haven't coded java since 2004. But I would still strongly push haproxy, as it essentially does precisely what you need, and will likely perform better. – izak Apr 22 '13 at 12:55
Threading is not the issue. If the load-balancer is not disconnected from the client it will hit the Operating System open file limit and new connections will be refused. Threading will also open a new file which will also cause the same problem. – App Work Apr 22 '13 at 14:07

How to create a software load balancer for socket connections in java

4 Answers4