9

I was helping someone with a question about outputting in C, and I was unable to answer this seemingly simple question I wanted to use the answer to (in my answer), that is:

What's the fastest way to output to a file in C / C++?

I've done a lot of work with prime number generation and mathematical algorithm optimization, using C++ and Java, and this was the biggest holdup for me sometimes - I sometimes need to move a lot to a file and fast.

Forgive me if this has been answered, but I've been looking on google and SO for some time to no avail.

I'm not expecting someone to do the work of benchmarking - but there are several ways to put to file and I doubt I know them all.

So to summarize,

What ways are there to output to a file in C and C++?

And which of these is/are the faster ones?

Obviously redirecting from the console is terrible. Any brief comparison of printf, cout, fputc, etc. would help.

Edit:

From the comments,

There's a great baseline test of cout and printf in: mixing cout and printf for faster output

This is a great start, but not the best answer to what I'm asking. For example, it doesn't handle std::ostreambuf_iterator<> mentioned in the comments, if that's a possibility. Nor does it handle fputc or mention console redirection (how bad in comparison)(not that it needs to)

Edit 2:

Also, for the sake of arguing my historical case, you can assume a near infinite amount of data being output (programs literally running for days on a newer Intel i7, producing gigabytes of text)

Temporary storage is only so helpful here - you can't buffer gigabytes of data easily that I'm aware.

Community
  • 1
  • 1
Plasmarob
  • 1,321
  • 12
  • 20
  • 1
    Have yo tried C++'s std::ostreambuf_iterator<>? –  Nov 14 '13 at 16:39
  • I haven't heard of that one. Thanks. I'll have to try and write some benchmarking tests with all of the methods I can find in a few days. – Plasmarob Nov 14 '13 at 16:42
  • possible duplicate of [mixing cout and printf for faster output](http://stackoverflow.com/questions/1924530/mixing-cout-and-printf-for-faster-output) – cbr Nov 14 '13 at 16:44
  • The answer is in that question. – cbr Nov 14 '13 at 16:44
  • 1
    Give this a try: call std::copy with instances of these iterators as arguments. This doesn't "interpret" the output as sending stuff to std::cout would usually do. A C-style solution may be faster, though. –  Nov 14 '13 at 16:45
  • That's why I asked - I handle data-structure-less mathematical code that's practically both languages, and I've heard C is faster. – Plasmarob Nov 14 '13 at 16:53
  • You should give memory mapped files a look. – StoryTeller - Unslander Monica Nov 14 '13 at 16:54
  • 1
    Try buffering your output then write to the file from the buffer in large blocks (use the block write functions, such as `ostream::write` or `fwrite()`). This should boost performance regardless of the language. Other tweaks, such as direct writing to the hard drive, may not have as big of an impact. – Thomas Matthews Nov 14 '13 at 17:01
  • Can you switch to a binary format? Gigabytes of text sounds like a poor representation. You can probably improve your performance from "really slow" to "just slow", but to make it actually fast, you'll want a different format. – Peter Nov 14 '13 at 17:38
  • Binary is a possibility. How much does that kind of formatting affect the implementation side? i.e., How would you efficiently implement binary file writing for say, a file of longs? (4 bytes)? And while we're on the subject, is there a fast random-access read? – Plasmarob Nov 14 '13 at 17:45
  • 1
    ostream::write() or fwrite(), for example. You call each one on an array of your data and tell it how many bytes you want to write. For random-access, ifstream::seekg() or fseek() will jump to whatever byte position you want, then istream::read() or fread() as usual. – Peter Nov 15 '13 at 13:31

4 Answers4

3

The functions such as fwrite, fprintf, etc. Are in fact doing a write syscall. The only difference with write is that these functions use a buffer to reduce the number of syscalls.

So, if I need to choose between fwrite, fprintf and write, I would avoid fprintf because it's a nice but complicated function that does a lot of things. If I really need something fast, I would reimplement the formating part myself to the bare minimum required. And between fwrite and write, I would pick fwrite if I need to write a lot of small data, otherwise write could be faster because it doesn't require the whole buffering system.

Maxime Chéramy
  • 17,761
  • 8
  • 54
  • 75
2

As far as I'm aware, the biggest bottleneck would be to write a character at a time (for example, using fputc). This is compared to building up a buffer in memory and dumping the whole lot (using fwrite). Experience has shown me that using fputc and writing individual characters is considerably slower.

This is probably because of hardware factors, rather than any one function being faster.

Phil_12d3
  • 388
  • 7
  • 18
  • So you're noting that sending larger chunks of data to a file is better? E.g. sending a couple dozen numbers / strings out with a stream instead of just one? – Plasmarob Nov 14 '13 at 16:50
  • 1
    In general, writing larger chunks is better. There is an overhead for each call to the output. Making 20 calls to write 1 character per call has 20 times the overhead as making 1 call to write 20 characters. – Thomas Matthews Nov 14 '13 at 16:56
  • 1
    @Plasmarob Yes. I had to load in a file that was 118 Mb. I started using fgetc, this took more than 15 minutes to load. I changed it to use fread and that went down to about 5 minutes. (I know you're talking about writing, rather than reading. But the same idea applies) – Phil_12d3 Nov 14 '13 at 16:58
2

The bottleneck in performance of output is formatting the characters.

In embedded systems, I improved performance by formatting text into a buffer (array of characters), then sending the entire buffer to output using block write commands, such as cout.write or fwrite. The functions bypass formatting and pass the data almost straight through.

You may encounter buffering by the OS along the way.

The bottleneck isn't due to the process of formatting the characters, but the multiple calls to the function.

If the text is constant, don't call the formatted output functions, write it direct: 

static const char  Message[] = "Hello there\n";
cout.write(&Message[0], sizeof(Message) - 1);  // -1 because the '\0' doesn't need to be written
Thomas Matthews
  • 56,849
  • 17
  • 98
  • 154
1

cout is actually slightly faster than printf because it is a template function, so the assembly is pre-compiled for the used type, although the difference in speed is negligible. I think that your real bottle neck isn't the call the language is making, but your hard-drives write rate. If you really want to go all the way with this, you could create a multi-thread or network solution that will store the data in a buffer, and then slowly write the data to the a hard-drive separate from the processing of the data.

aj.toulan
  • 1,341
  • 1
  • 16
  • 25
  • 1
    I thought template functions were slightly slower because of overhead. – Plasmarob Nov 14 '13 at 16:50
  • 2
    They take more time to compile, but once they are compiled it's just assembly. http://stackoverflow.com/questions/2442358/do-c-templates-make-programs-slow – aj.toulan Nov 14 '13 at 16:55
  • 1
    I would also pack your data, what I mean is to serialize it, or write it to the file as binary. So instead of writing "127" to a file which is 3 bytes. I would write 1 characters, (char)127 which is one byte of data, or 0111 1111 in binary. – aj.toulan Nov 14 '13 at 17:00