0

I have an application that reads large amount of data from the disk and also writes a lot. I'm trying to use direct io to improve the I/O performance.

Now I'm using libaio to implement the asynchronous IO, both for read and write. The preliminary result shows improvement in reading stage but great drop in writing stage.

Then I used strace to capture the run time behavior and here is the reading stage log(I have deleted some unrelated log):

8509  20:59:03.005920 io_submit(139876672323584, 16, {{pread, filedes:102, buf:0x7f36bf816000, nbytes:524288, offset:96468992}} <unfinished ...>
8509  20:59:03.007236 <... io_submit resumed> ) = 16 <0.000893>

and here is the writing stage log:

8098  20:47:40.219194 io_submit(140277578346496, 1, {{pwrite, filedes:116, str:"\177\362\215\264\252\360\240\306\377?\265\36/\215#%\304\0343\300\230\256\3550\374 k\316\v\225\327\""..., nbytes:524288, offset:24117248}}) = 1 <0.002457>

Generally the same io_submit API takes various time. Even though submitting 16 read requests at one time, it's still faster than submitting 1 write request.

So is it right? How can I optimize my write stage?

vincent_f
  • 83
  • 1
  • 5
  • It's not surprising that write is slower than read. Depending of a lot parameter (filesystem, technologie of support, ....), you can have a reallocation of memory (and so finding new place), maybe a slow erase phase before rewriting, .... In opposite, read is a "simple" access request. – Garf365 Aug 05 '16 at 09:05
  • What filesystem? What disk controller? What disk(s)? How are the disk(s) partitioned? Is there a RAID array? How is that configured? Are disk IO operations aligned with disk blocks? – Andrew Henle Aug 05 '16 at 11:35
  • Another way of using any async i/o on your OS which a lot of people forget about is to simply use memory mapped file sections. Truncate file B to the correct size, map in file A and file B, call memcpy(), do maps in sliding 10Mb windows if the file is big. On almost every OS out there this will perform an optimally efficient i/o pattern, an enormous amount of kernel tuning has been invested to ensure excellent performance. – Niall Douglas Aug 08 '16 at 07:36

1 Answers1

0

Did you open your file with O_DIRECT and were you writing a to a fully allocated file? If not then io_submit() can easily start taking a long time by essentially turning into a blocking call. See one of the answers to asynchronous IO io_submit latency in Ubuntu Linux for some of the things that must be done for io_submit() to perform properly.

Anon
  • 6,306
  • 2
  • 38
  • 56