10

According to this tutorial asynchronous disk file io can easily be achieved using AIO on linux, at least from a programming/api point-of-view. But before and after this tutorial I had read a lot of posts and articles that this either can not be done or you should use libevent with a patch and many other issues. Another thing was the loop that I should have waited for a signal, but based on this tutorial I can use a callback mechanism, which obviously makes AIO much easier to use.

Now, I am not a linux programmer even by a long shot I just wanted to find a straightforward way to support asynchronous disk file io on linux, learn it and add it to a async disk io library that I need for a personal project. Currently I'm using overlapped io on windows and io worker threads on non-windows platforms. Since the mutithreaded solution can be tricky, I wanted to replace it on linux with AIO.

SO, what is wrong with AIO as described in this tutorial? Is it performance? Is there a restriction on operations that can be done using AIO?

p.s. I don't care if the code will not be portable to other POSIX-compliant platforms, as long as it works on major linux distributions. And all I care about is regular disk file io.

Thanks.

p00ya00
  • 796
  • 1
  • 10
  • 20
  • 4
    AIO support on Linux is bad. Currently, AIO will bypass the buffer cache and perform disk IO directly, what may hurt performance badly. – lvella Dec 15 '11 at 00:43
  • can it be worse than the worker-threads solution? – p00ya00 Dec 15 '11 at 00:48
  • 1
    It depends on you usage. If all you do is to read or write a big file sequentially, it will perform well, but if you access small chunks randomly, it will certainly be worse. By the way, this worker-thread solution is what you get if you use the POSIX interface `aio_read()` etc. It is implemented with blocking threads on userspace by the glibc. – lvella Dec 15 '11 at 01:04
  • See this: http://stackoverflow.com/questions/5664105/buffered-asynchronous-file-i-o-on-linux – lvella Dec 15 '11 at 01:04
  • I want to use it in a web server. So you're saying that if I use my own worker-thread implementation which issues regular(cached) read/write calls in a thread, it would perform better than AIO/glibc? – p00ya00 Dec 15 '11 at 01:12
  • 1
    I say it would perform the same as POSIX AIO/glibc, and better than Linux native AIO (its interface is provided by libaio, with functions such `io_steup()` etc). – lvella Dec 15 '11 at 01:31
  • There is a good update to this question describing io_uring on https://stackoverflow.com/questions/13407542/is-there-really-no-asynchronous-block-i-o-on-linux – LustreOne Jul 23 '20 at 23:24

2 Answers2

14

The tutorial gives an overview of asynchronous I/O in general and talks about how there is kernel support for it. Then it goes on to talk about posix AIO (which is the standardized API for accessing asynchronous I/O), implying that using the posix AIO API on linux, will give you access to the kernel support for AIO. This is not the case.

On linux, there are really two separate AIO implementations:

  1. kernel AIO which uses io_submit() et al.) which is only supported in kernel 2.6 (or really 2.5 and there may be back-ported versions of it to 2.4.
  2. posix AIO which is a glibc feature, essentially unrelated to the kernel. It implements the posix API in terms of user-level threads making blocking disk I/O calls.

So, in short, if you already have a generic implementation of multiple threads for disk I/O, you might be better off using that than to use glibc's implementation (because you might have slightly more control over it).

If you're committed to actually use the io_submit() family of functions, you may have to do quite a lot of work to circumvent the restrictions on those functions.

kernel AIO requires your files to be opened with O_DIRECT. Which in turn requires all your file offsets, read and write sizes to be aligned to blocks on the disk. This is typically fine if you're just using one large file and you can make it work very similar to the page cache in the OS. For reading and writing arbitrary files at arbitrary offsets and lengths, it gets messy.

If you end up giving kernel AIO a shot, I would highly recommend looking into tying one or more eventfds to your iocbs so that you can wait on completions using epoll/select rather than having to block in io_getevents().

Arvid
  • 10,915
  • 1
  • 32
  • 40
3

The Linux implementation of POSIX AIO spawns a thread for every write you do. This is usually not good, and you're better off using your own worker threads to do the writes, so that you can control how many threads are in play. In other words, stick with what you have, AIO isn't going to buy you anything.

John Zwinck
  • 239,568
  • 38
  • 324
  • 436
  • 2
    The kernel AIO does not do this, but is less useful because it bypasses the buffer cache etc (which of course, is what databases want) – MarkR Dec 15 '11 at 07:33