1

I have a multi-threaded system in which a main thread has to wait in blocking state for one of the following 4 events to happen:

  1. inter-process semaphore (sem_wait())
  2. pthread condition (pthread_cond_wait())
  3. recv() from socket
  4. timeout expiring

Ideally I'd like a mechanism to unblock the main thread when any of the above occurs, something like a ppoll() with suitable timeout parameter. Non-blocking and polling is out of the picture due to the impact on the CPU usage, while having separate threads blocking on different events is not ideal due to the increased latency (one thread unblocking from one of the events should eventually wake up the main one).

The code will be almost exclusively compiled under Linux with gcc toolchain, if that helps, but some portability would be good, if at all possible.

Thanks in advance for any suggestion

  • Are you able to edit the code for the background threads? That is, can you add more code around wherever the semaphore and condition variables are modified in the background threads? – bnaecker Sep 29 '17 at 16:45
  • @bnaecker I have access to the intra-thread synchronization code (essentially, the threads that notify the pthread conditional variable) but not the inter-process semaphore, as the process that sem_post() on this semaphore I'm waiting for is 3rd party and outside my control. –  Sep 29 '17 at 16:58

1 Answers1

1

The mechanisms for waiting on multiple types of objects on Unix-like systems are not that great. In general, the idea is to, wherever possible, use file descriptors for IPC rather than multiple different IPC mechanisms.

From your comment, it sounds like you can edit or change the condition variable, but not the code that signals the semaphore. So what I'd recommend is something like the following.

Change the condition variable to either a pipe (for more portability) or an eventfd(2) object (Linux-specific). The notifying thread writes to the pipe whenever it wants to signal the main thread. This will allow you to select(2) or poll(2) or whatever in the main thread on both that pipe and the socket.

Because you're stuck with the semaphore, I think the best option would be to create another thread, whose sole purpose is to wait for the semaphore using sem_wait(), and then write to another pipe or eventfd(2) object when it is notified by whatever process is doing sem_post(). In the main thread, just add this other file descriptor to your select(2) set.

So you'll have three descriptors: one for the socket, one taking the place of the condition variable, and one which is written to when the semaphore is incremented. You can then wait on all three using your favorite I/O multiplexing method, and include directly whatever timeout you'd like.

bnaecker
  • 6,152
  • 1
  • 20
  • 33
  • Thanks. Replacing the pthread condition with a pipe is straightforward. We use the condition to signal the end of a job to the main thread, so a pipe can easily do this. However, I'm wondering, should I expect any performance (meaning latency) penalty in switching from pthread cond_wait to blocking recv on a pipe? –  Sep 30 '17 at 11:14
  • Never mind, found this very thorough analysis here: https://stackoverflow.com/questions/7979164/lowest-latency-notification-method-between-process-under-linux –  Sep 30 '17 at 14:09