How to create a single SQPOLL thread in io_uring for multiple rings (IORING_SETUP_SQPOLL)

Question

Is it possible to create a single SQPOLL (iou-sqp) thread that polls submit requests of multiple io_uring rings?

This questions comes from the desire to use multiple io_uring rings without making syscalls (entering kernel) when submitting I/O requests. In order to achieve this in case of a single ring, one creates an SQPOLL thread by passing the IORING_SETUP_SQPOLL flag to io_uring_setup() call. However, if multiple rings are created this way, multiple SQPOLL threads also get created (one thread for each ring). As a result we end up having several SQPOLL threads each busy polling a respective submit queue. Having a single SQPOLL thread would save CPU usage and in most of the cases would be enough to sustain the I/O load.

I tried to create a global uring file descriptor

static int RingFd;
static struct io_uring_params RingParams;
// ...
memset(&RingParams, 0, sizeof(RingParams));
RingParams.flags |= IORING_SETUP_SQPOLL;
RingParams.sq_thread_idle = 100;
RingFd = io_uring_setup(maxEvents, &RingParams);
if (RingFd < 0) {
    // ...
}
// ...

and mmap it to each uring

struct io_uring Ring;
int ret = io_uring_queue_mmap(RingFd, &RingParams, &Ring);
if (ret < 0) {
    // ...
}
// ...

but it doesn't work.

Answer 1

You can do this by using the IORING_ATTACH_WQ flag in combination with IORING_SETUP_SQPOLL. See the test case sq-poll-share in the liburing repo:

• https://github.com/axboe/liburing/blob/7ad5e52d4d2f91203615cd738e56aba10ad8b8f6/test/sq-poll-share.c

See also this conversation:

• https://github.com/axboe/liburing/issues/324

Relevants bits:

    for (i = 0; i < NR_RINGS; i++) {
        struct io_uring_params p = { };

        p.flags = IORING_SETUP_SQPOLL;
        if (i) {
            p.wq_fd = rings[0].ring_fd;
            p.flags |= IORING_SETUP_ATTACH_WQ;
        }
        ret = io_uring_queue_init_params(BUFFERS, &rings[i], &p);
        if (ret) {
            fprintf(stderr, "queue_init: %d/%d\n", ret, i);
            goto err;
        }
        /* no sharing for non-fixed either */
        if (!(p.features & IORING_FEAT_SQPOLL_NONFIXED)) {
            fprintf(stdout, "No SQPOLL sharing, skipping\n");
            return 0;
        }
    }