14

I need to create specialized HTTP server, for this I plan to use epoll sycall, but I want to utilize multiple processors/cores and I can't come up with architecture solution. ATM my idea is followng: create multiple threads with own epoll descriptors, main thread accepts connections and distributes them among threads epoll. But are there any better solutions? Which books/articles/guides can I read on high load architectures? I've seen only C10K article, but most links to examples are dead :( and still no in-depth books on this subject :(.

Thank you for answers.

UPD: Please be more specific, I need materials and examples (nginx is not an example because its too complex and has multiple abstraction layers to support multiple systems).

Daniel
  • 4,272
  • 8
  • 35
  • 48
  • 1
    Hey Daniel, I was wondering how this was coming along. I to am doing some research into the subject and I believe my concept of event driven servers may be a bit weak right now. To my knowledge, it seems that if we have an event driven back end ( say using epoll ) every function that gets called has to be non blocking... maybe it's my design but, every request that comes in makes a database call.. if that call is slow for whatever reason, all other client making requests at the same time also suffer due to waiting for the database response to finish. I can create a thread.. defeats purpose. – Begui Mar 09 '12 at 22:21
  • 4
    its not coming actually, its finished long time ago using libev :) some functions are still blocking in my app, just their block time is really small. but such things like database queries, i/o operations (especially when they're really intensive) should be non-blocking. in my case i'm using mongodb with its async-capable driver so i don't have any blockings while using db. i do have thread pools for things i couldn't make asynchronous (i.e. ImageMagick images processing and CSS/JS minimizers), but they're operated via queues and monitored by epoll (own queue implementation). – Daniel Mar 12 '12 at 18:13

3 Answers3

14

check libevent and libev sources. they're highly readable, and already a good infrastructure to use.

Also, libev's documentation has plenty of examples of several tried and true strategies. Even if you prefer to write directly to epoll(), the examples can lead to several insights.

Javier
  • 60,510
  • 8
  • 78
  • 126
  • from what i see libevent is single-threaded, but very efficient. can't find libevent+multithreading examples atm, but googling in progress... – Daniel Jan 14 '11 at 03:50
  • 1
    yes, both of them handles multithreading, check the multi-event-loop comments. – Javier Jan 14 '11 at 04:22
  • if you mean Steven Grimm's explanation on libevent home page its server is dead :( – Daniel Jan 14 '11 at 15:58
  • there's much more in the [libev docs](http://pod.tst.eu/http://cvs.schmorp.de/libev/ev.pod), it specifically mentions how to use in multithreaded applications, typically using 'the one loop per thread' design – Javier Jan 14 '11 at 17:15
  • libev is better documented as I see. did 15k requests with libev + multithread. thank you :) – Daniel Jan 15 '11 at 15:30
12

..my idea is followng: create multiple threads with own epoll descriptors, main thread accepts connections and distributes them among threads epoll.

Yes that's currently the best way to do this and it's how Nginx does it. The number of threads can be increased or decreased depending on load and/or the number of physical cores on the machine.

The trade-off between extra threads (more than the number of physical cores) and events is one of latency and throughput. Threads improve latency because they can execute pre-emptively but at the expense of throughput due to overhead incurred by context switching and thread creation/deletion. Events improve throughput but has the disadvantage that long-running code causes the entire thread to halt.

The second best is how Apache2 does it using a thread pool of blocking threads. No event processing here so the implementation is simpler and the pool means threads are not created and destroyed unnecessarily but it can't really compete with a well implemented thread/asynchronous hybrid like what you're trying to implement or Nginx.

The third best is asynchronous event processing alone like Lighttpd or Node.js. Well, it's the second best if you're not doing heavy processing in the server. But as mentioned earlier, a single long-running while loop blocks the entire server.

Fabian N.
  • 3,807
  • 2
  • 23
  • 46
slebetman
  • 109,858
  • 19
  • 140
  • 171
-3

Unless you have a terabit uplink and plan to service 10000 simultaneous connections off a single server, forget about epoll. It's just gratuitous non-portability; poll or even select will do just as well. Keep in mind that by the time terabit uplinks and such are standard, your server will also be sufficiently faster that you still won't need epoll.

If you're just serving static content, forget about threads too and use the Linux sendfile syscall. This too is nonstandard, but at least it offers huge real-world performance benefits.

Also note that other design decisions (especially excess complexity) will be much more of a factor in how much load your server can handle. For an example, just look how the modest single-threaded, single-process thttpd blows away Apache and friends in performance on static content -- and in my experience, even on traditional cgi dynamic content!

R.. GitHub STOP HELPING ICE
  • 208,859
  • 35
  • 376
  • 711
  • 1
    Servers haven't been significantly faster since 2005. 4GHz (or thereabouts) is apparently the limit. – slebetman Jan 14 '11 at 05:10
  • 1
    I still haven't seen a convincing argument for `epoll`, just a lot of `localhost` benchmarking and such. And I've seen huge reasons not to like it: all sorts of folks writing servers/apps that use `epoll` (and thus only work on Linux) when they only need to deal with 2-10 file descriptors... – R.. GitHub STOP HELPING ICE Jan 14 '11 at 05:47
  • I really do tests on localhost :) but still, i've been playing with libevent2 and it gave me ~10k r/s and 100% core load, so i'm thinking if I'll add threading it will be able do at least 20-25k requests (on 4 core box), so I still want to play with threading. – Daniel Jan 14 '11 at 15:56
  • @R..I think your comment really hits home, but all in all there's no shame in trying something new either. It's not more complex to use epoll than, say, poll or select and sometimes you have to take into account platform-specific bugs anyway (see libev documentation). It's also not that hard to generate some small platform specific abstractions for epoll, kqueue et cetera (fallback to poll when fast impl not available). See for example the nanomsg codebase: https://github.com/250bpm/nanomsg/tree/master/src/aio – Aktau Apr 27 '13 at 22:05
  • 3
    Though truth being told, one does not use epoll to listen on 10 sockets, but because it also works with timerfd and, more importantly eventfd. It makes one of the most stupid, ill-designed things you have to cope with under POSIX easy, straightforward, manageable. As a bonus, epoll is slightly faster and somewhat less awkward than `select` with its fd-set managing macros and obscure limits. Which, as you said, may not really matter, but it's no mistake either. – Damon Dec 06 '13 at 13:16
  • @Damon: All bogus. `timerfd` and `eventfd` work equally well with plain standard `poll`, but if you're using them you're doing something wrong anyway. `epoll` is definitely a lot slower than `poll` if you have transient file descriptors (since each add/remove is a syscall rather than a simple memory write). – R.. GitHub STOP HELPING ICE Dec 06 '13 at 16:17
  • Well, "slow" is debatable, and it sure depends on what the usage pattern is. Epoll is _designed_ and optimized for the not unusual case "modify descriptor set rarely, wait often". It is certainly more expensive to add/remove many descriptors (shame there is no `epoll_mctl` syscall). On the other hand, it is nice because it performs independently of the number of descriptors watched. Of course it can't be O(1) in respect of events (people sometimes expect that because common propaganda says "epoll is O(1)"), but it still beats select/poll in that respect. `poll` will return events much... – Damon Dec 06 '13 at 16:44
  • ... much alike `epoll` but still needs to transfer the descriptor set to wait, and `select` returns a fd-set, which is kind of ruling it out performance wise (unless you have a small limit on the fd-set such as 64 on Windows, but even with the much larger fd-set sizes you have under e.g. Linux, you may very well run against the limit -- poll and epoll do not have artificial limits, which is nice). – Damon Dec 06 '13 at 16:47
  • Until you get up to at least 1000 sockets, a syscall costs more than copying `poll`'s list of file descriptors from userspace; for `select` the threshold is probably more like 10000 (but of course `select` sucks in other ways and I recommend against all use of `select`). I think your "not unusual case" is actually the unusual case for most network traffic. The common cases are huge numbers of transient connections (HTTP) or small numbers of persistent connections. `epoll` makes more sense for something like IRC with massive numbers of persistent connections, not the usual case. – R.. GitHub STOP HELPING ICE Dec 06 '13 at 17:12