0

This open source scientific application written in c++ tends to generate lots of small writes in a short period of time (using ofstream). I am modifying the code such that the IO will not be as frequent.

It really only opens/writes to one or two files. What is the best way to monitor the IO frequency while the program is running? Or maybe to monitor the write frequency of certain files?

Edits: After I changed all the "endl" to "\n", strace shows the program is doing a lot of "writev" instead "write" with original code. Here is the strace output comparison.

Before:

write(4, "+\n", 2)                      = 2
write(4, ">>=@ABCEDBCFBEACDDCDDCBBCCDDCFAC"..., 77) = 77
write(3, "@C3A7DACXX140502:1:1314:6663:801"..., 37) = 37
write(3, "CATCAAACAAGATATGGATGTAGAACGCTCAG"..., 77) = 77
write(3, "+\n", 2)                      = 2
write(3, "?@>CCCDEDDFBDBCECCCEBAECCD@DDEDE"..., 77) = 77
write(4, "@C3A7DACXX140502:1:1314:6663:801"..., 37) = 37
write(4, "TTAGACAATGTAGTGAAATAGTTACTTTGGGG"..., 77) = 77
write(4, "+\n", 2)                      = 2
write(4, "==:BABBCDFBCEBECABBBE@B?DCDBDDCC"..., 77) = 77
write(3, "@C3A7DACXX140502:1:1314:6664:692"..., 36) = 36
write(3, "CAAAGACAGGTATGGAGACAGGAGAAGGGAGC"..., 77) = 77
write(3, "+\n", 2)                      = 2
write(3, ";=;3=6.?A'7*:;@AA=@@?C>.++68>),)"..., 77) = 77
write(4, "@C3A7DACXX140502:1:1314:6664:692"..., 36) = 36
write(4, "ACAAGATGTCTTCGGAGTTTCCGGGATAGCCA"..., 77) = 77

After:

writev(3, [{"\n@C3A7DACXX140502:3:1209:20544:1"..., 8186}, {"ACCTCCTCCTGCTTTCACCTATCCCGCTTCAC"..., 76}], 2) = 8262
writev(4, [{"\n@C3A7DACXX140502:3:1209:20544:1"..., 8186}, {"GATCCTCGTCAGTCCTGAAGGAGTGTCAGCTT"..., 76}], 2) = 8262
writev(3, [{"\n+\nBA@CDEEEDEFEDBBDCDBDBCDCB@DDB"..., 8148}, {"??>ABEEECADDBBABBDCEBDCDCBECDBC@"..., 76}], 2) = 8224
writev(4, [{"\n+\n>>;ABCD@BEDFCDCDECCECCE?DADCE"..., 8148}, {">>;ABABBDDECEDECECCCECBBECCDDDFD"..., 76}], 2) = 8224
writev(3, [{"\n@C3A7DACXX140502:3:1209:20618:6"..., 8184}, {"TTGGAAGGCCAGGTCCAGTAACCGGCCCCATT"..., 76}], 2) = 8260
writev(4, [{"\n@C3A7DACXX140502:3:1209:20618:6"..., 8184}, {"ATTAGTAATTTCAGTGCCTCCTCCATCTTTAG"..., 76}], 2) = 8260
writev(3, [{"\n+\n?@>CCDFEEDDDCBDBDEBBCDB@CDBBB"..., 8148}, {"B@<DDDDFCEDEBBDDBDBDC@EBBECDDCEC"..., 76}], 2) = 8224
writev(4, [{"\n+\n>=;?C@?CBCCEDEAEDCDDBDDBDCDEB"..., 8148}, {">><A@BED@DDBDECBCBECCECDBDCDBEED"..., 76}], 2) = 8224
writev(3, [{"\n@C3A7DACXX140502:3:1209:20684:3"..., 8180}, {"ACCCAAATGAGATCTGTGTGCCAATGTCAGTG"..., 76}], 2) = 8256
writev(4, [{"\n@C3A7DACXX140502:3:1209:20684:3"..., 8180}, {"TCATCTGTGAACTCCACCAAGTTTTGTGCCTC"..., 76}], 2) = 8256

Does that mean it is writing to buffer rather than flushing files?

Nasreddin
  • 1,509
  • 9
  • 31
  • 36
  • 1
    Note that ofstream class already does buffering internally, so by adding another layer of buffering you might not see much speedup: http://stackoverflow.com/questions/9425414/using-ofstream-for-buffered-output-to-gain-performance – Jeremy Friesner Apr 16 '15 at 19:38
  • The problem is we are seeing huge IOPS with this program. So we just wanted to write our own buffer to see if it makes a difference. – Nasreddin Apr 16 '15 at 19:42
  • Maybe first just set a larger buffer size on your existing ofstream... http://stackoverflow.com/questions/12757904/how-to-optimize-reading-and-writing-by-playing-with-buffer-size – Jeremy Friesner Apr 16 '15 at 20:02
  • Also check to see if your program is calling flush() on the ofstream object when it doesn't really need to. – Jeremy Friesner Apr 16 '15 at 20:03
  • What are you actually trying to achieve by doing this? It sounds like you "already have an idea of how to solve something", but you haven't actually described the original problem, just your suggested solution - aka "XY Problem"... – Mats Petersson Apr 16 '15 at 20:17
  • Why does it do a lot of small writes? It's no sense. ostream has a buffer mechanism that has to be bypassed on purpose. If it does, adding a new buffer will make it flush as it does in the upper level. – Luis Colorado Apr 17 '15 at 11:01

1 Answers1

2

IO tuning and performance improvement is difficult and time-consuming - you have to figure out your application's IO pattern(s) and match both code and hardware to that. And maybe even tune your file system, and you might even go so far as to select a file system based on your IO requirements in the first place. It's kind of like impedance matching electrical components for best performance - everything all the way from application code down to the disk heads themselves matter when you want to drive a system at its design limits.

And you can't drive a system at its design limits while simultaneously ignoring the design because of software abstractions like the C++ IO layer...

  1. Profile your application to find out if IO performance is the bottleneck. If it's not bottlenecked on IO, improving IO won't help application performance much if at all.

  2. Run the application under strace and see what's going on under the hood. Is the application doing a lot of seek()/lseek() system calls? Those invalidate any buffering done by the C++ runtime library and that would call for a revamped IO approach.

  3. If you do find your application needs to improve it's IO performance, you may need to redo the IO without using C++ IO routines - they're not fast. See here: std::fstream buffering vs manual buffering (why 10x gain with manual buffering)?.

  4. If you're just reading/writing large streams without seeking at all and don't re-read the data before it gets flushed out of the cache, the kernel page cache is only slowing you down. Use lower-level C-based IO routines, use page-aligned memory to read/write your data, and use the O_DIRECT flag on opening to perform direct IO and bypass the kernel page cache.

Community
  • 1
  • 1
Andrew Henle
  • 32,625
  • 3
  • 24
  • 56
  • It seems that the original code is using too many endl which forces it to flush every time. I changed them to "\n". When I look at strace output, now there is lots of "writev" instances rather than "write" with old code. Does that mean it is writing to buffer now? – Nasreddin Apr 17 '15 at 18:26
  • How many bytes get written per `writev()` system call? You should be able to see that in the strace output. How does the number of bytes compare to the amount of bytes written per `write()` call before your made your changes? – Andrew Henle Apr 17 '15 at 18:45
  • I've pasted the strace outputs from old and modiefied code. Which number is the bytes in writev, "8262"? It is a lot bigger than write which is around 77. Is this a good sign? – Nasreddin Apr 17 '15 at 18:53
  • It sure looks like it's using a 8192-byte buffer. I'd guess that when it's getting close to 8192, it'll flush the buffer when a call comes in that will fill the buffer. Try increasing the buffer size, to something like 128K – Andrew Henle Apr 17 '15 at 19:02
  • what is the best way to increase buffer size for ofstream, using pubsetbuf? – Nasreddin Apr 17 '15 at 19:18
  • That shoud work. Though you may find you're now pushing up against the limits of your filesystem and underlying hardware. – Andrew Henle Apr 17 '15 at 21:22
  • I tried to set buffer size to 200K with pubsetbuf, but it did not seem to work. [link](http://stackoverflow.com/questions/29777214/how-to-increase-buffer-size-of-ofstream) – Nasreddin Apr 21 '15 at 16:27
  • What do you mean? The larger buffer didn't help performance? – Andrew Henle Apr 21 '15 at 18:17
  • @ Andrew Henle strace did not show any change in IO behavior after I manually set the buffer size. See my other post in here [link](http://stackoverflow.com/questions/29777214/how-to-increase-buffer-size-of-ofstream) – Nasreddin Apr 21 '15 at 18:37