How to merge efficiently gigantic files with C#

Question

I have over 125 TSV files of ~100Mb each that I want to merge. The merge operation is allowed destroy the 125 files, but not the data. What matter is that a the end, I end up with a big file of the content of all the files one after the other (no specific order).

Is there an efficient way to do that? I was wondering if Windows provides an API to simply make a big "Union" of all those files? Otherwise, I will have to read all the files and write a big one.

Thanks!

Answer 1

So "merging" is really just writing the files one after the other? That's pretty straightforward - just open one output stream, and then repeatedly open an input stream, copy the data, close. For example:

static void ConcatenateFiles(string outputFile, params string[] inputFiles)
{
    using (Stream output = File.OpenWrite(outputFile))
    {
        foreach (string inputFile in inputFiles)
        {
            using (Stream input = File.OpenRead(inputFile))
            {
                input.CopyTo(output);
            }
        }
    }
}

That's using the Stream.CopyTo method which is new in .NET 4. If you're not using .NET 4, another helper method would come in handy:

private static void CopyStream(Stream input, Stream output)
{
    byte[] buffer = new byte[8192];
    int bytesRead;
    while ((bytesRead = input.Read(buffer, 0, buffer.Length)) > 0)
    {
        output.Write(buffer, 0, bytesRead);
    }
}

There's nothing that I'm aware of that is more efficient than this... but importantly, this won't take up much memory on your system at all. It's not like it's repeatedly reading the whole file into memory then writing it all out again.

EDIT: As pointed out in the comments, there are ways you can fiddle with file options to potentially make it slightly more efficient in terms of what the file system does with the data. But fundamentally you're going to be reading the data and writing it, a buffer at a time, either way.

Answer 2

Do it from the command line:

copy 1.txt+2.txt+3.txt combined.txt

or

copy *.txt combined.txt

Answer 3

Do you mean with merge that you want to decide with some custom logic what lines go where? Or do you mean that you mainly want to concatenate the files into one big one?

In the case of the latter, it is possible that you don't need to do this programmatically at all, just generate one batch file with this (/b is for binary, remove if not needed):

copy /b "file 1.tsv" + "file 2.tsv" "destination file.tsv"

Using C#, I'd take the following approach. Write a simple function that copies two streams:

void CopyStreamToStream(Stream dest, Stream src)
{
    int bytesRead;

    // experiment with the best buffer size, often 65536 is very performant
    byte[] buffer = new byte[GOOD_BUFFER_SIZE];

    // copy everything
    while((bytesRead = src.Read(buffer, 0, buffer.Length)) > 0)
    {
        dest.Write(buffer, 0, bytesRead);
    }
}

// then use as follows (do in a loop, don't forget to use using-blocks)
CopStreamtoStream(yourOutputStream, yourInputStream);

Answer 4

Using a folder of 100MB text files totalling ~12GB, I found that a small time saving could be made over the accepted answer by using File.ReadAllBytes and then writing that out to the stream.

        [Test]
        public void RaceFileMerges()
        {
            var inputFilesPath = @"D:\InputFiles";
            var inputFiles = Directory.EnumerateFiles(inputFilesPath).ToArray();

            var sw = new Stopwatch();
            sw.Start();

            ConcatenateFilesUsingReadAllBytes(@"D:\ReadAllBytesResult", inputFiles);

            Console.WriteLine($"ReadAllBytes method in {sw.Elapsed}");

            sw.Reset();
            sw.Start();

            ConcatenateFiles(@"D:\CopyToResult", inputFiles);

            Console.WriteLine($"CopyTo method in {sw.Elapsed}");
        }

        private static void ConcatenateFiles(string outputFile, params string[] inputFiles)
        {
            using (var output = File.OpenWrite(outputFile))
            {
                foreach (var inputFile in inputFiles)
                {
                    using (var input = File.OpenRead(inputFile))
                    {
                        input.CopyTo(output);
                    }
                }
            }
        }

        private static void ConcatenateFilesUsingReadAllBytes(string outputFile, params string[] inputFiles)
        {
            using (var stream = File.OpenWrite(outputFile))
            {
                foreach (var inputFile in inputFiles)
                {
                    var currentBytes = File.ReadAllBytes(inputFile);
                    stream.Write(currentBytes, 0, currentBytes.Length);
                }
            }
        }

ReadAllBytes method in 00:01:22.2753300

CopyTo method in 00:01:30.3122215

I repeated this a number of times with similar results.