4

I wrote trivial a node.js client/server pair to test local limits on concurrent connections. No data is sent between them: 10.000 clients connect, and wait.

Each time I run the test, I spawn a server and 10 clients that create 1000 connections each.

It takes a little over 2 seconds to reach ~8000 concurrent connections. Then it stops. No errors happen (on 'error' callbacks don't fire, close doesn't fire either). Connections "block", with no result or timeout.

I've already raised the max file descriptor limit (ulimit -n), and allowed more read/write memory to be consumed by the TCP stack via sysctl (net.ipv4.tcp_rmem and wmem).

What's the cap I'm hitting? How can I lift it?

-- EDIT --

Server program, with logging code stripped:

clients = []

server = net.createServer()

server.on 'connection', (socket) ->
    clients.push socket

server.listen 5050

Client (this runs n times):

sockets = []

for [1..num_sockets]
    socket = new net.Socket
    sockets.push socket
    socket.connect 5050

These are the system limits:

sysctl -w net.ipv4.ip_local_port_range="500 65535"  
sysctl -w net.ipv4.tcp_tw_recycle="1"               
sysctl -w net.ipv4.tcp_tw_reuse="1"                 

sysctl -w net.ipv4.tcp_rmem="4096 87380 16777216"   
sysctl -w net.ipv4.tcp_wmem="4096 16384 16777216"   

sysctl -w fs.file-max="655300"                      
sysctl -w fs.nr_open="3000000"                      

ulimit -n 2000000
salezica
  • 74,081
  • 25
  • 105
  • 166
  • What os you are using? Also if you can add your codes it would probably help to reproduce the behaviour. – jsalonen Jun 01 '14 at 07:38
  • Cannot reproduce the problem on Windows: 10k connections work just fine. Trying again with cygwin. – jsalonen Jun 01 '14 at 08:44
  • 1
    Could reproduce on cygwin. Possible duplicate of: http://stackoverflow.com/questions/17033631/node-js-maxing-out-at-1000-concurrent-connections - can you check out if those answers help! – jsalonen Jun 01 '14 at 08:48
  • @jsalonen the `posix` fix makes no difference. I didn't think it would, since I had set `nofile` (`hard` and `soft`) in `limits.conf` to a ridiculous number. Tried it anyway, but nope :( – salezica Jun 01 '14 at 17:22
  • Can you paste here what you get from `ulimit -n`? Also can you tell your node version. – jsalonen Jun 01 '14 at 17:42
  • Pasted all limits I'm aware of – salezica Jun 01 '14 at 18:10
  • Interesting. I got this to work in cygwin after calling `ulimit -n 10000`. Are you sure you are running the server with a user that has correct ulimit? – jsalonen Jun 01 '14 at 18:18
  • 100% sure, I even reinvoked `login` to get a fresh session after changing all the settings (so as to reflect changes in `/etc/security/limits.conf` as well) – salezica Jun 01 '14 at 18:21
  • If you only run 1 client with 10000 connections against the server does the error appear too? Or is it specific to running 10 clients with 1k connection each? – jsalonen Jun 01 '14 at 18:25
  • 1
    Fixed it. It was not a system issue: I limited the amount of simultaneous connecting sockets (not _connected_, _connecting_) to 10. As soon as a slot frees up, I start connecting another one. This fixes the issue. I'm sorry to have taken so much of your time – salezica Jun 01 '14 at 18:52

0 Answers0