2

When using POSIX asynchronous I/O, are there any differences between using O_DIRECT and AIO_RAW? Or should I/can I use both? We are working on a NoSQL database server and are looking at ways of making asynchronuous I/O more efficient on POSIX systems.

open(2):

O_DIRECT Try to minimize cache effects of the I/O to and from this file. In general this will degrade performance, but it is useful in special situations, such as when applications do their own caching. File I/O is done directly to/from user space buffers.

aiocb(5):

You may set the AIO_RAW flag bit in the aio_flags structure member when the asynchronous I/O is being done to a raw device partition. When the AIO_RAW flag bit is set, asynchronous I/O might possibly be more efficient.

  • Doesn't aio_write on Linux spawn a thread each time? This always seemed to me like a losing proposition--it may be better not to use AIO at all (at least on Linux until it's better). – John Zwinck Apr 14 '13 at 11:45
  • @JohnZwinck There is an interesting discussion on the subject here http://stackoverflow.com/questions/87892/what-is-the-status-of-posix-asynchronous-i-o-aio , it says `"Direct, unbuffered I/O is only really useful for transactional databases, and those tend to write their own threads or processes to manage their disk I/O."` - which is our scenario as we are working on a transactional NoSQL database server product. We manage our own user thread pool. –  Apr 14 '13 at 11:53
  • I'm not saying don't try to do direct, unbuffered I/O. I'm saying consider not using AIO. And that's pretty much what the page you linked says too: avoid AIO. Use O_DIRECT if you want, but never AIO_RAW, because never AIO. :) – John Zwinck Apr 14 '13 at 11:56
  • @JohnZwinck I think that as long as we are managing our own user thread pool we should be ok; the nice thing with POSIX AIO is that it works on every file system (ext2, ext3, jfs, xfs and nfs) and POSIX systems, this way we don't have to write the same code many times to make a uniform device interface. –  Apr 14 '13 at 12:09

1 Answers1

0

The questioner never specified which operating systems they were supporting (one link was to a Linux man page the other to a SCO one) so it's hard to answer this without speculation.

At the time of writing Linux doesn't supports AIO_RAW because glibc's struct aiocb doesn't have an aio_flags member (https://www.gnu.org/software/libc/manual/html_node/Asynchronous-I_002fO.html ) and the in kernel AIO (which is NOT POSIX) doesn't support a AIO_RAW value. So:

are there any differences between using O_DIRECT and AIO_RAW? Or should I/can I use both?

Yes because not all operating systems with POSIX AIO support (such as Linux) allow you to set AIO_RAW and given that there will definitely be situations where you can't use both. Further, on some operating systems such as Linux O_DIRECT can work for files inside filesystems AND on devices whereas AIO_RAW is intended only for a "raw device". Judging from the dt source code it looks like AIO_RAW was only supported on HP-UX and SCO Unix so it's probably of limited use with other OSes.

Anon
  • 6,306
  • 2
  • 38
  • 56