8

I am working on a threaded network server using epoll (edge triggered) and threads and I'm using httperf to benchmark my server.

So far, it's performing really well or almost exactly at the rate the requests are being sent. Until the 1024 barrier, where everything slows down to around 30 requests/second.

Running on Ubuntu 9.04 64-bit.

I've already tried:

  • Increasing the ulimit number of file descriptors, successfully. It just doesn't improve the performance above 1024 concurrent connections.

    andri@filefridge:~/Dropbox/School/Group 452/Code/server$ ulimit -n
    20000

I am pretty sure that this slow-down is happening in the operating system as it happens before the event is sent to epoll (and yes, I've also increased the limit in epoll).

I need to benchmark how many concurrent connections my program can handle until it starts to slow down (without the operating system interfering).

How do I get my program to run with more than 1024 file descriptors?

This limit is probably there for a reason, but for benchmarking purposes, I need it gone.

Update

Thanks for all your answers but I think I've found the culprit. After redefining __FD_SETSIZE in my program everything started to move a lot faster. Of course ulimit also needs to be raised, but without __FD_SETSIZE my program never takes advantage of it.

Andrioid
  • 3,362
  • 4
  • 27
  • 31
  • 3
    If you've resolved the issue yourself, you can "Answer" your own question, and "Accept" your own answer, to mark this question resolved. – ephemient May 11 '09 at 22:43
  • @android facing slow response time issue. i have raised ulimit but not getting where to increase this __FD_SETSIZE . please tell which file needs to be edited – Steeve Apr 13 '17 at 14:30

3 Answers3

7

Thanks for all your answers but I think I've found the culprit. After redefining __FD_SETSIZE in my program everything started to move a lot faster. Of course ulimit also needs to be raised, but without __FD_SETSIZE my program never takes advantage of it.

Andrioid
  • 3,362
  • 4
  • 27
  • 31
  • Using an FD_SET with fd's beyond __FD_SETSIZE causes data that happens to be after the FD_SET to be overwritten, which can cause plenty of hard-to-debug grief. I am a little curious why you are using an FD_SET with epoll (it would make sense for select() or poll()...) – Lance Richardson May 12 '09 at 11:42
  • Because it made a difference. Using httperf to bomb my server it stalled at 1000/s without my FD_SETSIZE changes, stalls at 5000/s at the moment with httperf complaining about fd-unavailable. What exactly happens in the program when I get too many connections is that the epoll listen file descriptor just stops getting events. Nothing locks down, my event loop still runs. Which points me towards the operating systems or any underlying libraries causing the limit. – Andrioid May 12 '09 at 15:18
  • It's not necessaryly epoll that is not receiving the file descriptors fast enough. It is quite possible that httperf (that uses select) is causing this limitation, I'll update my answer when I know more. – Andrioid May 13 '09 at 06:08
  • I tried FD_SETSIZE instead of __FD_SETSIZE first, thanks for your answer! – Prof. Falken Nov 02 '10 at 14:03
  • @android facing slow response time issue. i have raised ulimit but not getting where to increase this __FD_SETSIZE . please tell which file needs to be edited – Steeve Apr 13 '17 at 14:31
4

Please see the C10K problem page. It contains an in-depth discussion on how to achieve the '10000 simultaneous connections' goal, while maintaining high-performance and managing to serve each client.

It also contains information on how to increase the performance of your kernel when handling a large number of connections at once.

ASk
  • 4,157
  • 1
  • 18
  • 15
-4

Just don't.

Yes, I mean that.

If you need to increase the file descriptors, there's a hidden bug in your code. Hunt it down instead of treating its symptoms. Remember to close file descriptors when you're done.

asdf
  • 249
  • 3
  • 10