Differences in poll() between Linux and OS X when pollfd is changed on another thread

Question

I'm trying to get libwebsockets running in a multithreaded environment on OS X. I couldn't trigger sending Data from a different thread than the main service thread. On libwebsocket docs it was implied this should be possible (demo code, mailinglist). So I dug into the code and found the problem in the poll() function.

It seems that poll() is behaving differently concerning the struct pollfd that is given as parameter. libwebsockets is relying on the possibility to change the fds.event fields while poll() is active. This is working fine on Linux but is not working on OS X.

I wrote a small test program to demonstrate the behaviour:

#include <unistd.h>
#include <netdb.h>
#include <poll.h>
#include <iostream>
#include <thread>

#define PORT "3490"

struct pollfd    fds[1];
bool connected = false;

void main_loop() {
    int sockfd, new_fd; 
    struct addrinfo hints, *servinfo, *p;
    socklen_t sin_size;
    int yes=1;
    char s[INET6_ADDRSTRLEN];
    int rv;

    memset(&hints, 0, sizeof hints);
    hints.ai_family = AF_INET;
    hints.ai_socktype = SOCK_STREAM;
    hints.ai_flags = AI_PASSIVE; 

    if ((rv = getaddrinfo(NULL, PORT, &hints, &servinfo)) != 0) {
        fprintf(stderr, "getaddrinfo: %s\n", gai_strerror(rv));
        return;
    }

    for(p = servinfo; p != NULL; p = p->ai_next) {
        if ((sockfd = socket(p->ai_family, p->ai_socktype, p->ai_protocol)) == -1) {
            perror("server: socket");
            continue;
        }

        if (setsockopt(sockfd, SOL_SOCKET, SO_REUSEADDR, &yes, sizeof(int)) == -1) {
            perror("setsockopt");
            exit(1);
        }

        if (bind(sockfd, p->ai_addr, p->ai_addrlen) == -1) {
            close(sockfd);
            perror("server: bind");
            continue;
        }

        break;
    }

    freeaddrinfo(servinfo);

    if (p == NULL)  {
        fprintf(stderr, "server: failed to bind\n");
        exit(1);
    }

    if (listen(sockfd, 10) == -1) {
        perror("listen");
        exit(1);
    }

    printf("server: waiting for connections...\n");

    new_fd = accept(sockfd, NULL, &sin_size);
    if (new_fd == -1) {
        perror("accept");
        return;
    }

    fds[0].fd = new_fd;
    fds[0].events = POLLIN;
    connected = true;

    printf("event is %i\n", fds[0].events);
    int ret = poll(fds, 1, 5000);
    printf("event is %i\n", fds[0].events); //expecting 1 on Mac and 5 on Linux

    if (send(new_fd, "Hello, world!\n", 14, 0) == -1)
        perror("send");

    close(new_fd); 
    close(sockfd);
}

void second_thread()
{
    while(connected == false){}
    sleep(1);
    fds[0].events = POLLIN|POLLOUT;
    printf("set event to %i\n", fds[0].events);
}

int main() {

    std::thread t1(main_loop);
    std::thread t2(second_thread);

    t1.join();
    t2.join();

    return 0;
}

Compile on OS X using clang++ -std=c++11 -stdlib=libc++ -o poll poll.cpp and on Linux using g++ -std=c++11 -pthread -o poll poll.cpp

The program starts listening on port 3490. If you connect to it (e.g. using netcat localhost 3490) it will poll for input on the main thread and try to change the event flags in the second thread. It will exit after 5 seconds.

The output on OS X:

server: waiting for connections...
event is 1
set event to 5
event is 1

The output on Linux:

server: waiting for connections...
event is 1
set event to 5
event is 5

So my question is: is there any documentation available that explains this behavior? Is it safe what libwebsockets is doing in expecting that it is legal to change fds.events while poll is active? I couldn't find any details about it in the manpages (OS X, Linux).

Answer 1

You seem to say, at first, that you found some documentation that claims that this is supported and defined behavior. I'd be curious to know where you read that, because I am unable to find anything in either the Linux man page for poll(2), nor in the POSIX man page for poll() that documents that a different thread can actually change the values in the event array argument that another thread passed to poll(), and have the different thread's changes actually take effect in the original thread's poll() call, irrespective of any issues relating to memory barriers, and such.

Both man pages appear to be completely silent, to me, on this subject matter. They do not indicate whether this is expected, supported, or defined behavior; or whether this is not a supported or defined behavior.

The proposition that a different thread can modify the parameters to a system call issued by another thread, after -- AFTER -- the other thread has already entered the syscall, seems rather counter-intertuitive to me. If this is supported behavior, I would expect it to be explicitly documented, and I can't find any reference to it in the Linux or the POSIX man pages.

Having said that: even if I limit the scope of my software to Linux, even if I don't need to care about other platforms; given the absence of any documentation of this, and even if my testing showed the Linux kernel implementing poll(2) this way, I would not expect to have any guarantees that some future kernel version will continue to behave this way. I would not be able to rely on this behavior, except on the specific kernel build I tested this with.

So, to answer your question: the only documentation that's authoritative on this topic are the man pages in question. They do not explicitly document this as legal behavior; and although they do not explicitly say that this is illegal behavior either, for the reasons stated above, I would consider this to be unsupported, undefined behavior.