Transactional handling of text files on Windows

Question

I have multiple Windows programs (running on Windows 2000, XP and 7), which handle text files of different formats (csv, tsv, ini and xml). It is very important not to corrupt the content of these files during file IO. Every file should be safely accessible by multiple programs concurrently, and should be resistant to system crashes. This SO answer suggests using an in-process database, so I'm considering to use the Microsoft Jet Database Engine, which is able to handle delimited text files (csv, tsv), and supports transactions. I used Jet before, but I don't know whether Jet transactions really tolerate unexpected crashes or shutdowns in the commit phase, and I don't know what to do with non-delimited text files (ini, xml). I don't think it's a good idea to try to implement fully ACIDic file IO by hand.

What is the best way to implement transactional handling of text files on Windows? I have to be able to do this in both Delphi and C#.

Thank you for your help in advance.

EDIT

Let's see an example based on @SirRufo's idea. Forget about concurrency for a second, and let's concentrate on crash tolerance.

1. I read the contents of a file into a data structure in order to modify some fields. When I'm in the process of writing the modified data back into the file, the system can crash.

2. File corruption can be avoided if I never write the data back into the original file. This can be easily achieved by creating a new file, with a timestamp in the filename every time a modification is saved. But this is not enough: the original file will stay intact, but the newly written one may be corrupt.

3. I can solve this by putting a "0" character after the timestamp, which would mean that the file hasn't been validated. I would end the writing process by a validation step: I would read the new file, compare its contents to the in-memory structure I'm trying to save, and if they are the same, then change the flag to "1". Each time the program has to read the file, it chooses the newest version by comparing the timestamps in the filename. Only the latest version must be kept, older versions can be deleted.

4. Concurrency could be handled by waiting on a named mutex before reading or writing the file. When a program gains access to the file, it must start with checking the list of filenames. If it wants to read the file, it will read the newest version. On the other hand, writing can be started only if there is no version newer than the one read last time.

This is a rough, oversimplified, and inefficient approach, but it shows what I'm thinking about. Writing files is unsafe, but maybe there are simple tricks like the one above which can help to avoid file corruption.

UPDATE

Open-source solutions, written in Java:

• Atomic File Transactions: article-1, article-2, source code
• Java Atomic File Transaction (JAFT): project home
• XADisk: tutorial, source code
• AtomicFile: description, source code

Answer 1

How about using NTFS file streams? Write multiple named(numbered/timestamped) streams to the same Filename. Every version could be stored in a different stream but is actually stored in the same "file" or bunch of files, preserving the data and providing a roll-back mechanism... when you reach a point of certainty delete some of the previous streams.

Introduced in NT 4? It covers all versions. Should be crash proof you will always have the previous version/stream plus the original to recover / roll-back to.

Just a late night thought.

http://msdn.microsoft.com/en-gb/library/windows/desktop/aa364404%28v=vs.85%29.aspx

Answer 2

What you are asking for is transactionality, which is not possible without developing yourself the mechanism of a RDBMS database according to your requirements:

"It is very important not to corrupt the content of these files during file IO"

Pickup a DBMS.

Answer 3

See a related post Accessing a single file with multiple threads However my opinion is to use a database like Raven DB for these kind of transactions, Raven DB supports concurrent access to same file as well as supporting batching on multiple operations into a single request. However everything is persisted as JSON documents, not text files. It does support .NET/C# very well, including Javascript and HTML but not Delphi.

Answer 4

First of all this question has nothing to do with C# or Delphi. You have to simulate your file structure as if it is a database.

Assumptions;

• Moving of files is a cheap process and Op System guarantees that the files are not corrupted during move.

• You have a single directory of files that need to be processed. (d:\filesDB*.*)

• A Controller application is a must.

Simplified Worker Process;

-initialization

1. Gets a processID from the Operating system.

2. Creates directories in d:\filesDB

   d:\filesDB\<processID>
   d:\filesDB\<processID>\inBox
   d:\filesDB\<processID>\outBox
   

-process for each file

1. Select file to process.
2. Move it to the "inBox" Directory (ensures single access to file)
3. Open file
4. Create new file in "outBox" and close it properly
5. Delete file in "inBox" Directory.
6. Move newly created file located in "OutBox" back to d:\filesDB

-finallization

1. remove the created directories.

Controller Application

Runs only on startup of the system, and initializes applications that will do the work.

1. Scan d:\filesDB directory for subdirectories,
2. For each subDirectory 2.1 if File exists in "inBox", move it to d:\filesDB and skip "outBox". 2.2 if File exists in "outBox", move it to d:\filesDB 2.3 delete the whole subDirectory.
3. Start each worker process that need to be started.

I hope that this will solve your problem.

Answer 5

You are creating a nightmare for yourself trying to handle these transactions and states in your own code across multiple systems. This is why Larry Ellison (Oracle CEO) is a billionaire and most of us are not. If you absolutely must use files, then setup an Oracle or other database that supports LOB and CLOB objects. I store very large SVG files in such a table for my company so that we can add and render large maps to our systems without any code changes. The files can be pulled from the table and passed to your users in a buffer then returned to the database when they are done. Setup the appropriate security and record locking and your problem is solved.

Answer 6

Ok, you are dead - unless you can drop XP. Simple like that.

Since POST-XP Windows supports Transactional NTFS - though it is not exposed to .NET (natively - you can still use it). This allows one to roll back or commit changes on a NTFS file system, with a DTC even in coordination with a database. Pretty nice. XP, though - no way, not there.

Start at Any real-world, enterprise-grade experience with Transactional NTFS (TxF)? as a starter. The question there lists a lot of ressources to get you started on how to do it.

Note that this DOES have a performance overhead - obviously. It is not that bad, though, unless you need a SECOND transactional resource, as there is a very thin kernel level transaction coordinator there, transactions only get promoted to full DTC when a second ressource is added.

For a direct link - http://msdn.microsoft.com/en-us/magazine/cc163388.aspx has some nice information.