File I/O performance is too good to be true?

Question

I ran a simple file I/O performance test. I wrote 1GB data to a file and measure the time elapsed. The result shows the write time takes only about 0.45sec and performance is over 17Gbps. I know this is impossible but I can't find anything wrong in my test code. Below is my test routine. I can see correct file in d:\a.bin.

#include <stdio.h>
#include <stdlib.h>
#include <time.h>

int main()
{
    char *ioBuf;
    int i, io_buf_size = 1024 * 1024;
    unsigned int io_amt = 0;
    FILE *fp;
    clock_t start, end;
    double elapsed;

    ioBuf = (char *)malloc(1024 * 1024 * sizeof(char));

    for (i = 0; i < io_buf_size; i++) {
        ioBuf[i] = i % 255;
    }

    if ((fp = fopen("d:\\a.bin", "wb")) == NULL) {
        printf("open file fail, err  \n");
        return 1;
    }

    start = clock();
    for (i = 0; i < 1024; i++) {
        io_amt += fwrite(ioBuf, sizeof(char), io_buf_size, fp);
        if (fflush(fp) != 0) printf("flushing buffer failed!\n");
    }
    end = clock();
    elapsed = (double)(end - start) / CLOCKS_PER_SEC;
    printf("fwrtie %dGB takes : %f sec\n", io_amt / (1024 * 1024 * 1024), elapsed);

    fclose(fp);
    free(ioBuf);
    return 0;
}

Your file is in the RAM, the OS automatically caches your files system to prevent useless physical accesses and speed up writes. — Guillaume F., Aug 24 '17 at 16:42
This routine calls fflush() for every 1MB write. Doesn't it force data to be written to physical media? — chungae9ri, Aug 25 '17 at 12:33
`fflush()` only sends the data from the stream to the file buffer, the OS can still perform optimizations unless you disable them. Look at this answer for more info : https://stackoverflow.com/a/16781818/5444618 — Guillaume F., Aug 26 '17 at 02:08
[Don't cast the result of `malloc` in C](https://stackoverflow.com/q/605845/995714) — phuclv, Aug 26 '17 at 03:20

BeeOnRope · Accepted Answer · 2017-08-26T19:07:12.327

The OS generally caches writes in RAM and writes them out only after some period usually measured in seconds (depending on the exact OS configuration). Beyond that even your disk drive may have a RAM cache¹.

If you write enough data (e.g., more than the amount of RAM you have), you'll see a dramatic slowdown at some point as the OS now has to synchronously write out file data to make room for your new writes.

Note that this OS-level caching is different than the caching that might be done by the standard library when you do an fwrite - you are writing 1 MB chunks so probably aren't seeing standard library caching, and in any case when you fclose() the file those buffers will be flushed.

To measure the true speed you have a variety of options. Assuming you are on a POSIX system like Linux you can use fsync on the integer file descriptor that you can get from fileno. This will flush both the OS buffers and instruct the disk to write out any non-durable disk buffers. Or just abandon the C library f* functions entirely and use the OS-level routines to open the file O_DIRECT.

Finally, as mentioned in the comments, the contents of your file is a simple repeating pattern (with a period of 255 bytes). File systems using compression or de-duplication might easily store this file in a small faction of its nominal size, leading to an "out of the this world" apparent write speed, even with fsync used properly. To avoid this possibility, write random data.

¹ A cache on spinning disks of a few 10s of MB is common, but this is less common on SSDs.

Also the drive(s) will be doing its own caching too. And if it's a journaled fs, "writing a file" is followed up sometime later with flushing the journal to the fs proper. All very complicated! — bazza, Aug 26 '17 at 06:50
Yes, but at least the drive caching is typically limited to a few dozen MB, and `fsync` should cause a flush at both the OS and drive level. — BeeOnRope, Aug 26 '17 at 17:09
It's also possible to get "incredible" performance writing to a file system with native compression. Write something like all zeros and you can get some insane "raw" IO numbers even when the data is safely synced all the way to the disks. Even the question's data pattern will be highly compressible, and any halfway-decent compression algorithm only has to increment a counter. — Andrew Henle, Aug 26 '17 at 17:15
@AndrewHenle - good point, I updated the question to reflect that and bazza's comment about disk buffers. — BeeOnRope, Aug 26 '17 at 19:08

File I/O performance is too good to be true?

1 Answers1