4

If I'm writing a simple text log file from multiple processes, can they overwrite/corrupt each other's entries?

(Basically, this question Is file append atomic in UNIX? but for Windows/NTFS.)

Community
  • 1
  • 1
Gkakk McJkakk
  • 105
  • 1
  • 4

4 Answers4

13

You can get atomic append on local files. Open the file with FILE_APPEND_DATA access (Documented in WDK). When you omit FILE_WRITE_DATA access then all writes will ignore the the current file pointer and be done at the end-of file. Or you may use FILE_WRITE_DATA access and for append writes specify it in overlapped structure (Offset = FILE_WRITE_TO_END_OF_FILE and OffsetHigh = -1 Documented in WDK).

The append behavior is properly synchronized between writes via different handles. I use that regularly for logging by multiple processes. I do write BOM at every open to offset 0 and all other writes are appended. The timestamps are not a problem, they can be sorted when needed.

Yakeen
  • 2,142
  • 1
  • 17
  • 21
  • Wow again, Windows isn't so bad but it tries hard to hide the features. Does this work with all Desktop programs, IoCreateFileSpecifyDeviceObjectHint looks a bit scary and i never did Device Driver codeing. – Lothar Jun 27 '12 at 00:10
  • Yes it does, WriteFile works well. See http://msdn.microsoft.com/en-us/library/windows/desktop/aa365747%28v=vs.110%29.aspx. There is example in community additions. – Yakeen Jul 18 '12 at 12:14
  • 1
    Do you have any reference for "The append behavior is properly synchronized between writes via different handles"? – avakar Jul 18 '12 at 12:20
  • 1
    The IRPs are stored in queue (priority Vista+) and the operations are locked. That's implemented differently than on Linux. Windows has file operations locking (e.g. deleted file exists till you have a handle to it). Linux append is guaranteed to be atomic only for data up to page size. – Yakeen Jul 18 '12 at 12:55
  • Note that the blurb about the mentioned DDK API says that the access flags you need are `FILE_APPEND_DATA` and nothing else. If you OR in other access flags, such as read or delete, you will not get the atomic append. (Setting the offset to `~0ULL` in the `OVERLAPPED` struct works regardless of access flags though.) – asveikau May 17 '13 at 18:02
  • I'm finding this doesn't work in Java. See http://stackoverflow.com/questions/22326703 – Rich Mar 11 '14 at 13:13
  • @Rich, it depends on implementation, on Windows you need to use FILE_APPEND_DATA flag or use the special constants, Posix require O_APPEND flag. Although Java has bool value for append parameter looking into its source code following comment explains why it does not work: [/* append is ignored */](http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7-b147/sun/nio/ch/FileDispatcherImpl.java#FileDispatcherImpl) – Yakeen Jul 01 '14 at 20:13
  • Thanks -- that answers my question. If you want to add it as an answer to that question, I'll accept it over there. – Rich Jul 02 '14 at 14:41
  • @Yakeen, your comments about the implementation are invaluable, I have some vague memory that some implementations have a upper byte-count limit on the atomic writes, but I always forget Linux vs Windows, thank you for comparing the two. – Cameron Jan 23 '15 at 18:34
  • @Cameron, you're right, O_APPEND guarantee atomic writes only up to page size. – Yakeen Sep 06 '15 at 19:54
2

Even if append is atomic (which I don't believe it is), it may not give you the results you want. For example, assuming a log includes a timestamp, it seems reasonable to expect more recent logs to be appended after older logs. With concurrency, this guarantee doesn't hold - if multiple processes are waiting to write to the same file, any one of them might get the write lock - not just the oldest one waiting. Thus, logs can be written out of sequence.

If this is not desirable behaviour, you can avoid it by publishing logs entries from all processes to a shared queue, such as a named pipe. You then have a single process that writes from this queue to the log file. This avoids the conccurrency issues, ensures that logs are written in order, and works when file appends are not atomic, since the file is only written to directly by one process.

mdma
  • 56,943
  • 12
  • 94
  • 128
  • Thanks for your response and alternative solution; I've set your reply as "accepted answer" for the suggestion of using a single process with named pipes – Gkakk McJkakk Jun 13 '10 at 14:31
1

From this MSDN page on creating and opening Files:

An application also uses CreateFile to specify whether it wants to share the file for reading, writing, both, or neither. This is known as the sharing mode. An open file that is not shared (dwShareMode set to zero) cannot be opened again, either by the application that opened it or by another application, until its handle has been closed. This is also referred to as exclusive access.

and:

If you specify an access or sharing mode that conflicts with the modes specified in the previous call, CreateFile fails.

So if you use CreateFile rather than say File.Open which doesn't have the same level of control over the file access, you should be able to open a file in such a way that it can't get corrupted by other processes.

You'll obviously have to add code to your processes to cope with the case where they can't get exclusive access to the log file.

ChrisF
  • 134,786
  • 31
  • 255
  • 325
0

No it isn't. If you need this there is Transactional NTFS in Windows Vista/7.

Axel Gneiting
  • 5,293
  • 25
  • 30
  • Thanks for your response, I will keep that option in mind for the future (unfortunately this program is for Windows 2003 so I can't use TxF) – Gkakk McJkakk Jun 13 '10 at 14:30
  • Axel, The append is atomic and not only on NTFS, see my answer. – Yakeen Jan 02 '12 at 08:46
  • 1
    Microsoft strongly recommends developers investigate utilizing [alternatives](http://msdn.microsoft.com/en-us/library/windows/desktop/hh802690%28v=vs.85%29.aspx) to Transactional NTFS (or in some cases, investigate other alternatives) rather than adopting an API platform which may not be available in future versions of Windows. – Aaron Stainback Dec 29 '12 at 10:18
  • I don't think that was already the case in 2010. But thanks for the information. – Axel Gneiting Jan 05 '13 at 16:11