How to wake up a pthread which is sleeping in recvmsg()?

Question

I have a little Unix daemon which uses pthreads. One thread runs in a loop, reading network packets using recvmsg. When the daemon receives a signal, a flag is set telling all the threads to break out of their loops and exit. However, the "listening" thread never checks the flag until the next packet arrives and recvmsg returns, which can take some time.

Delivering a SIGINT to the "listening" thread using pthread_kill makes it break out of recvmsg, but it also invokes the signal handler which processes breaks sent by pressing Ctrl+C at the console. That is unwanted. Another way to force recvmsg to return early is to close the socket which it is listening on, but I think there must be a better way. Do you know what it is?

This daemon is used on Linux only, if it makes a difference.

Answer 1

pthread_cancel may be what you're looking for. Read the manual pages carefully though and don't overuse it.

Answer 2

Three ideas:

Idea #1: Why not another signal? The signals SIGUSR1 and SIGUSR2 exist solely for user-defined signalling. This might be the easiest approach if you're already setting up signal handlers and generally comfortable with their use.

Idea #2: Send a dummy packet as others suggested.

Idea #3: If you really want to avoid sending a packet, try using non-blocking I/O such that recvmsg won't block at all (see MSG_DONTWAIT).

Instead, call epoll | poll | select to block on the socket recv event AND another file descriptor.

So a loop that looks like this:

while(1){
   recvmsg();
   do_stuff();
}

Becomes:

while(1){
   wait_for_events();
   if( /* FD used for cancellation is readable */ )
      break;
   else
      recvmsg();
      do_stuff();
}

Where wait_for_events is either select, poll, or epoll_wait. Set up the wait list to include both your socket AND an additional fd such as a pipe, unix-domain socket, or POSIX message queue (which is a file in Linux).

To cancel the socket reader thread(s), do a write on the cancellation file object. Then have reader(s) check to see if that fd is able to be read when waking up from poll/epoll/select.

You should be able to use one such file to wake ALL threads if there are multiple instances of the thing processing recvmsg, just make sure you use level-triggering not edge-triggering if using epoll.

If you're not sure which of those three calls to use, I suggest starting with poll and only use select if you already know it really well or epoll if you know you need its scalability.

Answer 3

If this is UDP, a common approach I use is to set the exit condition flag and then just send a 1 byte packet to force the recvfrom/recvmsg call to return.

Alternatively, you can set a timeout value (e.g. 1 second) on your socket SO_RCVTIMEO. When the socket wait times out, you can check the exit condition. If the exit condition is not set, then call recvmsg again.

https://stackoverflow.com/a/4182564/104458

Answer 4

Install an empty signal handler to an unused POSIX realtime signal, SIGRTMIN+0 to SIGRTMAX-0, and use pthread_kill() to send that signal to the thread blocked in recvmsg(). Realtime signals are queued (reliable) whereas standard signals are not. This means that interrupting two different threads at the same time works if you use a realtime signal, but may fail if using a normal signal (such as SIGUSR1).

It is the delivery of a signal to a signal handler, that interrupts a library function or a system call -- even if the body of the signal handler function is empty. (Assuming, of course, that you install the signal handler using sigaction() without the SA_RESTART flag. See man 7 signal, under "Interruption of system calls and library functions by signal handlers" for details.)

---

The one problem you may encounter with this approach (using a signal to interrupt a blocking I/O function) is when the signal is raised early, before the target thread is actually blocking on the I/O function. That window cannot really be avoided, you just have to deal with it.

Personally, I like to use a dedicated thread maintaining timeout structures, something like

struct timeout {
    pthread_t        who;
    struct timespec  when;  /* Using CLOCK_MONOTONIC */
    volatile int     state;
};

that each thread can acquire and release -- I've found particularly useful that a thread can hold multiple timeouts at the same time -- and a simple atomic function to check the state (to be used in e.g. loop conditions).

The trick is that when the timeout initially triggers, it is not removed, due to the aforementioned problematic time window. Instead, I like to mark the timeout "elapsed", and just rearm it a millisecond of a few milliseconds later. This repeats, until the target thread releases the timeout. (Each time a timeout is triggered, I do send a POSIX realtime signal to the target thread, with an empty signal handler function installed.)

This way, even if your recv()/send() loops occasionally miss the signal, they will be notified very soon afterwards.

You will also want to install a thread cleanup handler that releases all timeouts owned by the thread, if you ever cancel threads. Otherwise you may "leak" timeout structures.

If you use clock_gettime() and pthread_cond_timedwait(), to wait for a new timeout entries (added by other threads) or an existing timeout to elapse, the timeout management thread will be very lightweight and not consume much memory or CPU time at all. Note that you can use CLOCK_MONOTONIC, if you create the new-timeouts-added condition variable with an attribute set that has that clock selected via pthread_condattr_setclock().