Synchronize with sigev_notify_function()

Question

I would like to read (asynchronously) BLOCK_SIZE bytes of one file, and the BLOCK_SIZE bytes of the second file, printing what has been read to the buffer as soon as the respective buffer has been filled. Let me illustrate what I mean:

// in main()
    int infile_fd = open(infile_name, O_RDONLY); // add error checking
    int maskfile_fd = open(maskfile_name, O_RDONLY); // add error checking
    char* buffer_infile = malloc(BLOCK_SIZE); // add error checking
    char* buffer_maskfile = malloc(BLOCK_SIZE); // add error checking
    struct aiocb cb_infile;
    struct aiocb cb_maskfile;
    // set AIO control blocks
    memset(&cb_infile, 0, sizeof(struct aiocb));
    cb_infile.aio_fildes = infile_fd;
    cb_infile.aio_buf = buffer_infile;
    cb_infile.aio_nbytes = BLOCK_SIZE;
    cb_infile.aio_sigevent.sigev_notify = SIGEV_THREAD;
    cb_infile.aio_sigevent.sigev_notify_function = print_buffer;
    cb_infile.aio_sigevent.sigev_value.sival_ptr = buffer_infile;

    memset(&cb_maskfile, 0, sizeof(struct aiocb));
    cb_maskfile.aio_fildes = maskfile_fd;
    cb_maskfile.aio_buf = buffer_maskfile;
    cb_maskfile.aio_nbytes = BLOCK_SIZE;
    cb_maskfile.aio_sigevent.sigev_notify = SIGEV_THREAD;
    cb_maskfile.aio_sigevent.sigev_notify_function = print_buffer;
    cb_maskfile.aio_sigevent.sigev_value.sival_ptr = buffer_maskfile;

and the print_buffer() function is defined as follows:

void print_buffer(union sigval sv)
{
    printf("%s\n", __func__);
    printf("buffer address: %p\n", sv.sival_ptr);
    printf("buffer: %.128s\n", (char*)sv.sival_ptr);
}

By the end of the program I do the usual clean up, i.e.

// clean up
    close(infile_fd); // add error checking
    close(maskfile_fd); // add error checking
    free(buffer_infile);
    printf("buffer_inline freed\n");
    free(buffer_maskfile);
    printf("buffer_maskfile freed\n");

The problem is, every once in a while buffer_inline gets freed before print_buffer manages to print its contents to the console. In a usual case I would employ some kind of pthread_join() but as far as I know this is impossible since POSIX does not specify that sigev_notify_function must be implemented using threads, and besides, how would I get the TID of such thread to call pthread_join() on?

Sounds like you're firing off your AIO calls and not waiting for them to complete before cleaning up. Don't do that. But that's just a guess because you haven't posted a complete example. — Andrew Henle, Jan 06 '22 at 19:18
I already know that this is the problem. How can I fix it? Or maybe this approach shouldn't be used in this particular case? — AJB, Jan 06 '22 at 20:33
This is probably a good start: [**lio_listio: How to wait until all requests complete?**](https://stackoverflow.com/questions/14777284/lio-listio-how-to-wait-until-all-requests-complete) IME, asynchronous IO on Linux is usually not worth the code complexity. Last I looked, glibc just spawns separate threads to do the IO in user space. Things may have changed since then, but without actual kernel and filesystem support for asynchronous IO calls (such as Solaris' `kaio()` system call), you get little to no performance boost at the cost of some really complex code. — Andrew Henle, Jan 06 '22 at 21:06

pilcrow · Answer 1 · 2022-01-12T14:46:47.720

Don't do it this way, if you can avoid it. If you can, just let process termination take care of it all.

Otherwise, the answer indicated in Andrew Henle's comment above is right on. You need to be sure that no more sigev_notify_functions will improperly reference the buffers.

The easiest way to do this is simply to countdown the number of expected notifications before freeing the buffers.

Note: your SIGEV_THREAD function is executed in a separate thread, though not necessarily a new thread each time. (POSIX.1-2017 System Interfaces §2.4.2) Importantly, you are not meant to manage this thread's lifecycle: it is detached by default, with PTHREAD_CREATE_JOINABLE explicitly noted as undefined behavior.

As an aside, I'd suggest never using SIGEV_THREAD in robust code. Per spec, the signal mask of the sigev_notify_function thread is implementation-defined. Yikes. For me, that makes it per se unreliable. In my view, SIGEV_SIGNAL and a dedicated signal-handling thread are much safer.

Synchronize with sigev_notify_function()

1 Answers1