Multi-Threaded Event Handling in C++

Question

When we want to design a generic Event Handling (I/O Demultiplexing) or reactor pattern model. The underlying system call we use is "select" or "poll". But both of the system call is not thread safe on common set of FD's. Can not be used in Multithreading environment.

What would be the better approach to handle events with multiple threads across multiple I/O's.

The 1 way I can see is, main thread recv's any events and push into a shared queue of thread pool. But worker thread can not send the data over the I/O causes synchronization problems. Also has memory overflow disadvantage.

All possible suggestions are welcome. Thanks in Advance.

I would investigate boost asio, it wraps the low-level details for you and you work with a nicer interface which can handle various io "events" already and allow you to "post" your own events on to the "queue" to be processed... — Nim, Aug 22 '12 at 09:34
You'd use one thread for polling, and assign the actual I/O onto worker threads (out of a thread pool). — Jörgen Sigvardsson, Aug 22 '12 at 09:34

score 1 · Answer 1 · answered Aug 23 '12 at 07:27

Most Unixes provide more scalable alternatives to select/poll that can be used in multithreaded environments:

Linux has epoll
BSDs have kqueue
Solaris has event completion ports

But getting it right in a multithreaded environment can be tricky, so you might want to look at existing abstraction layers like boost.asio

On the other hand, boost.asio does introduce some non-negligible overhead - I have collected some information about that and an alternative abstraction for epoll/kqueue at http://nginetd.cmeerw.org

Thanks.. I will go through these refferences. – Naveen Aug 23 '12 at 10:33 — Naveen, Aug 23 '12 at 10:33

Andrew Tomazos · Answer 2 · 2012-08-23T04:45:27.780

0

One way is to have one input thread, one output thread, several worker threads, and two blocking queues.

Input thread parses input messages and places them in queue 1. Worker threads all wait on queue 1, process a message and places any output they have in queue 2. Output thread waits on queue 2 and serializes the output from it.

InputThread:
    Loop:
        M = ReadNextMessage
        Q1.push(M)

AddOutput(O):
    Q2.push(O)

WorkerThread:
    Loop:
        M = Q1.pop
        ProcessMessage(M) using AddOutput as needed

OutputThread:
    Loop:
        O = Q2.pop
        WriteOutput(O)

However I don't know what you mean that select isn't threadsafe ? All system calls are technically threadsafe

It's very old but still very interesting is an article called "The CY10K problem" by a guy called Kegel..

http://www.kegel.com/c10k.html

It discusses different ways to structure a "multi io" program on Linux and the pros and cons of each.

edited Aug 23 '12 at 04:45

answered Aug 23 '12 at 03:50

Andrew Tomazos

66,139
40
186
319

`select` is not threadsafe with the same fd. You should have read the manuel. – J-16 SDiZ Aug 23 '12 at 03:52
@J-16SDiZ Can you explain in what way? Do you mean if the fd_set is outside of the threads and then multiple threads call select using that fd_set? – Adrian Cornish Aug 23 '12 at 03:56
@J-16SDiZ: Incorrect. select is thread safe as stated in POSIX manual: http://stackoverflow.com/questions/12084704/select-on-same-fd-from-multiple-sockets – Andrew Tomazos Aug 23 '12 at 05:47
@AndrewTomazos-Fathomling, it wasn't back in UNIX98. Many unix (e.g. AIX) explicitly saying this don't work. – J-16 SDiZ Aug 23 '12 at 05:57

score 0 · Answer 3 · answered Aug 23 '12 at 04:41

0

Check out the "Producer - Consumer problem", this is a very good starting point for thread-safe problems. It can also be expanded to multiple producers and multiple consumers.

answered Aug 23 '12 at 04:41

Alex Calugarescu

838
5
9

Multi-Threaded Event Handling in C++

3 Answers3