fwrite() performance well below disk capacity

Question

I have a dynamically allocated array of a struct with 17 million elements. To save it to disk, I write

fwrite(StructList, sizeof(Struct), NumStructs, FilePointer)

In a later step I read it with an equivalent fread statement, that is, using sizeof(Struct) and a count of NumStructs. I expect the resulting file will be around 3.5 GB (this is all x64).

Is it possible instead to pass sizeof(Struct) * NumStructs as the size and 1 as the count to speed this up? I am scratching my head as to why the write operation could possibly take minutes on a fast computer with 32 GB RAM (plenty of write cache). I've run home-brew benchmarks and the cache is aggressive enough that 400 MB/sec for the first 800 MB to 1 GB is typical. PerfMon shows it is consuming 100% of one core during the fwrite.

I saw the question here so what I'm asking is, whether there is some loop inside fwrite that can be "tricked" to go faster by telling it to write 1 element of size n*s as opposed to n elements of size s.

EDIT

I ran this twice in release mode and both times I gave up waiting. Then I ran it in debug mode knowing that typically the fwrite operations take way longer. The exact size of the data to be written is 4,368,892,928 bytes. In all three cases, PerfMon shows two bursts of disk write activity about 30 seconds apart, after which the CPU goes to 100% of one core. The file is at that point 73,924,608 bytes. I have breakpoints on either side of the fwrite so I know that's where it's sitting. It certainly seems that something is stuck but I will leave it running overnight and see.

EDIT

Left this overnight and it definitely hung in fwrite, the file never went past 70 MB.

Answer 1

This is definitely a problem with fwrite (I tried both VS2012 and 2010).

Starting with a standard C++ project, I changed only the setting to use multi-byte character set, x64 target, and the multithreaded debug version of the standard library in a static link.

The following code succeeds (no error checking for conciseness):

#define _CRT_SECURE_NO_WARNINGS
#include <stdio.h>
#include <stdlib.h>

int main()
{
    FILE *fp;
    long long n;
    unsigned char *data;

    n = 4LL * 1024 * 1024 * 1024 - 1;

    data = (unsigned char *)malloc(n * sizeof(unsigned char));

    fp = fopen("T:\\test.bin", "wb");

    fwrite(data, sizeof(unsigned char), n, fp);

    fclose(fp);
}

In the debug version on my machine, the program finishes in about 1 minute (the malloc takes only a few seconds, so this is mostly fwrite), consuming on average 30% CPU. PerfMon shows the write occurs entirely at the end is a single "flash" of 4 GB (write cache).

Change the - 1 to a + 1 in the assignment of n and you reproduce the problem: instantaneous 100% CPU usage and nothing is ever written. After several minutes, the size of the file was still 0 bytes (recall in my actual code it manages to dump 70 MB or so).

This is definitely a problem in fwrite, as the following code can write the file just fine:

int main()
{
    FILE *fp;
    long long n;
    long long counter = 0;
    long long chunk;
    unsigned char *data;

    n = 4LL * 1024 * 1024 * 1024 + 1;

    data = (unsigned char *)malloc(n * sizeof(unsigned char));

    fp = fopen("T:\\test.bin", "wb");

    while (counter < n)
    {
        chunk = min(n - counter, 100*1000);
        fwrite(data+counter, sizeof(unsigned char), chunk, fp);
        counter += chunk;
    }

    fclose(fp);
}

On my machine, this took 45 seconds instead of 1 minute. CPU usage is not constant, it comes in bursts, and the reported IO writes are more distributed than in the "single chunk" method.

I would be really surprised if the increase in speed is false (that is, due to caching) because I've done tests before writing several files containing all the same data vs files containing randomized data and the reported write speeds (with caching) are the same. So I'm willing to bet that at least this implementation of fwrite does not like huge chunks passed to it at a time.

I also tested fread to read immediately after closing the file for write in the 4 GB+1 case and it returns in a timely manner - a few seconds at most (no real data here so I didn't check it).

EDIT

I ran some tests with the chunk-writing method and the single fwrite call of a 4 GB-1 file (the largest size which both methods can do). Running the program several times (with code such that the file is opened, written with multiple fwrite calls, closed, then opened again, written in a single call, and closed), there is no question the chunk-writing method returns faster. At worst case it returns in 68% of the time it takes for the single call and at best I got just 20%.

Answer 2

This is not a problem with fwrite but intended (though admittedly uncool) behavior:

The fwrite() function shall write, from the array pointed to by ptr, up to nitems elements whose size is specified by size, to the stream pointed to by stream. For each object, size calls shall be made to the fputc() function, taking the values (in order) from an array [...]

So basically, by using fwrite correctly without cheating, you are requesting billions of calls to fputc.
With the above requirement in mind, it's clear how you have to cheat in order to make it work properly, too.