QFile: multiple handles to same physical file do not write all data

Question

I was wondering how QFile behaves when multiple handles are opened to the same file (using C++ in Visual Studio 2013 on Windows 7), so I wrote the following little program:

  QFile file("tmp.txt");
  file.open(QIODevice::WriteOnly | QIODevice::Truncate);
  QTextStream ts(&file);
  ts << "Hallo\n";

  QFile file2("tmp.txt");
  file2.open(QIODevice::WriteOnly | QIODevice::Append);
  QTextStream ts2(&file2);
  ts2 << "Hallo 2\n";

  file.close();

  ts2 << "Hello again\n";

  file2.close();.

This produces the following output in file tmp.txt:

Hallo 2
Hello again

So the first Hallo statement got lost. If I do a ts.flush() right after the ts << "Hallo\n" this does not happen, which makes me think that the statement got lost in the internal buffers of QString or it was overwritten by the subsequent output statements. However, I want to use QFile in a logging framework, so I don't want to always flush, as this would decrease the performance.

I also tried the same thing with std::basic_ostream<char> instead of QFile:

  std::basic_ofstream<char> file;
  file.open("tmp.txt", std::ios_base::out | std::ios_base::ate | std::ios_base::app);
  file << "Hallo\n";

  std::basic_ofstream<char> file2;
  file2.open("tmp.txt", std::ios_base::out | std::ios_base::ate | std::ios_base::app);
  file2 << "Hallo 2\n";

  file.close();

  file2 << "Hello again\n";

  file2.close();

which outputs as I would expect:

Hallo
Hallo 2
Hello again

So what is the problem with the QFile example? Is QFile not intended to be used with multiple handles pointing to the same file or what is going on here exactly? I thought that my use case is quite a common one, so I'm a bit surprised to find this behaviour. I couldn't find more specifics in the Qt documentation. I've read here that Qt opens the file in shared mode, so this shouldn't be a problem.

I eventually want to use QFile for logging (where access to the function that does the actual writing is of course synchronized), but this little example worries me that some log statements might get lost on the way. Do you think it would be better to use STL streams instead of QFile?

Edit As it was pointed out, std::endl causes a flush, so I changed the STL example above to only use \n which according to here does not cause a flush. The behavior described above is unchanged, though.

Just tested it without `std::endl` - same behaviour (post edited) — kafman, Feb 15 '16 at 14:46

score 1 · Accepted Answer · answered Feb 15 '16 at 15:48

I seems you want it both ways.

If you want several write buffers and don't want to flush them, it's hard to be sure of having all the writes in the file, and in the right order. Your small test with std::basic_ostream is not a proof: will it work with larger writes ? Will it work on other OSes ? Do you want to risk your process for a (yet unproven) speed gain ?

AchmadJP · Answer 2 · 2016-02-15T15:06:20.490

0

 QFile file("tmp.txt");
  file.open(QIODevice::WriteOnly | QIODevice::Truncate);
  QTextStream ts(&file);
  ts << "Hallo\n";

  QFile file2("tmp.txt");
  file2.open(QIODevice::Append);
  QTextStream ts2(&file2);
  ts2 << "Hallo 2\n";

  file.close();

  ts2 << "Hello again\n";

  file2.close();

Try this, i changed it so the Truncate not getting invoke by WriteOnly. Silly me didn't read that one. {UPDATE}

QIODevice::WriteOnly    0x0002  The device is open for writing. Note that this mode implies Truncate.
QIODevice::Truncate 0x0008  If possible, the device is truncated before it is opened. All earlier contents of the device are lost.

Reading Source : http://doc.qt.io/qt-5/qiodevice.html#OpenModeFlag-enum

edited Feb 15 '16 at 15:06

answered Feb 15 '16 at 14:59

AchmadJP

893
8
17

Same behaviour - why would the order of the arguments for the binary or be of relevance? – kafman Feb 15 '16 at 15:02
Sorry, see my update. Can you try that? ReadWrite invoke Write which invoke Truncate. – AchmadJP Feb 15 '16 at 15:03
Find it, here : http://doc.qt.io/qt-5/qiodevice.html#OpenModeFlag-enum There you go. – AchmadJP Feb 15 '16 at 15:04
Ok, I changed it to `file.open(QIODevice::WriteOnly)` and `file2.open(QIODevice::Append)` but still the same behavior – kafman Feb 15 '16 at 15:13

score 0 · Answer 3 · answered Feb 15 '16 at 15:00

There are several suspicious things going on. For starters, you are introducing two levels of buffering.

First and foremost QTextStream has an internal buffer, that you can flush by calling flush on it.
_{Second, QFile is also buffering (or, better, it's using the buffered APIs from your library -- fopen, fwrite, and so on). Pass QIODevice::Unbuffered to open to make it use the unbuffered APIs (open, write, ...).}

Now since this is terribly error prone, QTextStream::flush actually flushes also the underlying file device.

Also, you're passing WriteOnly | Append which doesn't make sense. It's only one of the two.

However, note that your writes may still interleave. POSIX.1-2013 says that

A write is atomic if the whole amount written in one operation is not interleaved with data from any other process. This is useful when there are multiple writers sending data to a single reader. Applications need to know how large a write request can be expected to be performed atomically. This maximum is called {PIPE_BUF}.

(On Windows I have no idea).

Regarding the two levels of buffering: what you say makes sense, but using `QTextStreams` is actually how it is done in the Qt documentation, too ... http://doc.qt.io/qt-5/qfile.html. FWIW, I experience the same behavior if I don't use a stream, but directly `file.write()`. Regarding the flags: you're right, I changed it to only use `QIODevice::Append` and added QIODevice::Unbuffered, but it ddid not change the result. — kafman, Feb 15 '16 at 15:20
So if I use `QIODevice::Unbuffered` and directly operate on the file with `file.write()` then it works as you suggested. However, as pointed out in my original post, I would like to find away that doesn't use flushing (or in this example, that does use buffers) in order to speed up performance. So is the STL just superior? — kafman, Feb 15 '16 at 15:25

QFile: multiple handles to same physical file do not write all data

3 Answers3