1

I'm using the Linux kernel AIO through libaio, and I have to submit the next reading operation before the previous one was completed. The problem is that io_submit() blocks for some time and, as I can deduce from the interval, it waits the previous operation to be completed.

I know that I can enqueue a several operations with a single io_submit(), but it is not an option for me, because I don't know how exactly the next read operation would like when it's already a time to submit the first one.

Is it working like that only for me, or for everyone? In the second case, may I ask if I'm looking for something feasible, or I have to fallback to a threaded model?

sangram parmar
  • 8,462
  • 2
  • 23
  • 47
vdudouyt
  • 843
  • 7
  • 14
  • `io_submit` will block under weird, unpredictable conditions (which is why I'm not using it). I once asked around and got an answer like "of course, it has to work that way, there is a limited-size request queue". Happens that e.g. doing one large request may be broken into several smaller ones, so... queue full and blocks. The threaded glibc async I/O implementation works better but be aware that it will spawn threads on demand (if that is not acceptable, use your own pool of workers). – Damon Jul 02 '15 at 08:03
  • Also note that if you don't turn off buffering, kernel aio will run synchronously anyway. A patch (by some Indian guy) which would allow for buffered async I/O has been around for a decade or so, but was turned down on the base "nobody needs that". – Damon Jul 02 '15 at 08:05
  • In my use of it, I've seen it blocking for tens of microseconds, occasional blip in the 100-200us range, but nothing much higher. Still, it was enough to put it in it's own thread. If yours is blocking longer than that, make sure you opened with O_DIRECT and that you are using 512-byte aligned memory buffers (and lengths). Also, if you're doing it to a file on a filesystem, make sure the filesystem supports it. It's tricky! – Mike Andrews Jul 02 '15 at 17:21
  • I'm using ext4, which is supporting it. And did you tried to do a second io_submit() while the first operation is not completed? – vdudouyt Jul 03 '15 at 02:28
  • @Minoru, yes, I am submitting I/O's concurrently with io_submit(), though not on ext4. – Mike Andrews Jul 07 '15 at 21:24
  • @gubblebozer, may I ask you to share some of your code with me? Just to make sure that I'm doing everything right. – vdudouyt Jul 08 '15 at 02:35
  • @Minoru I can't share this piece of code. Plus, it's actually quite a lot of code surrounding how the structures are built up. I can, however, take a look at yours -- that is, if you can share it. Unfortunately, the io_submit interface tends to fail in exactly the way you're experiencing. It "works", but the I/O's block. – Mike Andrews Jul 08 '15 at 15:14
  • 1
    I just stumbled across this again after 2 years and noticed mention of "ext4". Note that ext4 does not **really** support aio at all (or the other way around, aio doesn't support ext4). Even if you do everything "correctly" (`O_DIRECT`, which is a major anti-optimization, and aligned reads), you may still have `io_submit` block. If, for whatever reason, filesystem metadata needs be read, this will block inside `io_submit`, and you're out of luck. Sadly, blocking and waiting for one seek is about the same as waiting for several megabytes transferred anyway, so this renders aio kinda pointless. – Damon Jun 08 '17 at 22:26
  • I believe if the underlying (block) device driver (and/or hardware) queues are full - the submission can/will block until there are resources available. – Brad Jul 23 '18 at 17:55

1 Answers1

0

Frustratingly there are lots of reasons why io_submit may block including:

  • You're doing buffered I/O
  • You're doing I/O into a filesystem and your submission is queued behind a synchronous operation.

It's known that ext4 and AIO may not be the best mix:

Blocking during io_submit on ext4, on buffered operations, network access, pipes, etc. [...] AIO access to a file on a filesystem like ext4 is partially supported: if a metadata read is required to look up the data block (ie if the metadata is not already in memory), then the io_submit call will block on the metadata read. Certain types of file-enlarging writes are completely unsupported and block for the entire duration of the operation.

(extract is from a document that was called AIOUserGuide)

See the asynchronous IO io_submit latency in Ubuntu Linux question for other detailed answers.

Anon
  • 6,306
  • 2
  • 38
  • 56