Merging huge (2GB) XMLs in memory (without any memory exceptions)

Question

I would like a C# code that optimally appends 2 XML strings. Both of them are of same schema. I tried StreamReader / StreamWriter; File.WriteAllText; FileStream The problem I see is, it uses more than 98% of physical memory thus results in out of memory exception.

Is there a way to optimally merge without getting any memory exceptions? Time is not a concern for me.

If making it available in memory is going to be a problem, then what else could be better? Saving it on File system?

Further Details: Here is my simple program: to provide better detail

static void Main(string[] args)
        {
            Program p = new Program();
            XmlDocument x1 = new XmlDocument();
            XmlDocument x2 = new XmlDocument();
            x1.Load("C:\\XMLFiles\\1.xml");
            x2.Load("C:\\XMLFiles\\2.xml");
            List<string> files = new List<string>();
            files.Add("C:\\XMLFiles\\1.xml");
            files.Add("C:\\XMLFiles\\2.xml");
            p.ConsolidateFiles(files, "C:\\XMLFiles\\Result.xml");
            p.MergeFiles("C:\\XMLFiles\\Result.xml", x1.OuterXml, x2.OuterXml, "<Data>", "</Data>");
            Console.ReadLine();

        }

        public void ConsolidateFiles(List<String> files, string outputFile)
        {
            var output = new StreamWriter(File.Open(outputFile, FileMode.Create));
            output.WriteLine("<Data>");
            foreach (var file in files)
            {
                var input = new StreamReader(File.Open(file, FileMode.Open));
                string line;
                while (!input.EndOfStream)
                {
                    line = input.ReadLine();
                    if (!line.Contains("<Data>") &&
                        !line.Contains("</Data>"))
                    {
                        output.Write(line);
                    }
                }
            }
            output.WriteLine("</Data>");
        }
        public void MergeFiles(string outputPath, string xmlState, string xmlFederal, string prefix, string suffix)
        {
            File.WriteAllText(outputPath, prefix);
            File.AppendAllText(outputPath, xmlState);
            File.AppendAllText(outputPath, xmlFederal);
            File.AppendAllText(outputPath, suffix);
        }

XML Sample: <Data> </Data> is appended at the beginning & End

XML 1: <Sections> <Section></Section> </Sections>

XML 2: <Sections> <Section></Section> </Sections>

Merged: <Data> <Sections> <Section></Section> </Sections> <Sections> <Section></Section> </Sections> </Data>

Answer 1

Try this, a stream based approach which avoids loading all the xml into memory at once.

    static void Main(string[] args)
    {
        List<string> files = new List<string>();
        files.Add("C:\\XMLFiles\\1.xml");
        files.Add("C:\\XMLFiles\\2.xml");
        ConsolidateFiles(files, "C:\\XMLFiles\\Result.xml");
        Console.ReadLine();
    }

    private static void ConsolidateFiles(List<String> files, string outputFile)
    {
        using (var output = new StreamWriter(outputFile))
        {
            output.WriteLine("<Data>");
            foreach (var file in files)
            {
                using (var input = new StreamReader(file, FileMode.Open))
                {
                    while (!input.EndOfStream)
                    {
                        string line = input.ReadLine();
                        if (!line.Contains("<Data>") &&
                            !line.Contains("</Data>"))
                        {
                            output.Write(line);
                        }
                    }
                }
            }
            output.WriteLine("</Data>");
        }
    }

An even better approach is to use XmlReader (http://msdn.microsoft.com/en-us/library/system.xml.xmlreader(v=vs.90).aspx). This will give you a stream reader designed specifically for xml, rather than StreamReader which is for reading general text.

Answer 2

Take a look here The answer given by Teoman Soygul seems to be what you're looking for.

Answer 3

This is untested, but I would do something along these lines using TextReader and TextWriter. You do not want to read all of the XML text into memory or store it in a string, and you do not want to use XElement/XDocument/etc. anywhere in the middle.

using (var writer = new XmlTextWriter("ResultFile.xml")
{
    writer.WriteStartDocument();
    writer.WriteStartElement("Data");
    using (var reader = new XmlTextReader("XmlFile1.xml")
    {
        reader.Read();
        while (reader.Read())
        {
            writer.WriteNode(reader, true);
        }
    }
    using (var reader = new XmlTextReader("XmlFile2.xml")
    {
        reader.Read();
        while (reader.Read())
        {
            writer.WriteNode(reader, true);
        }
    }
    writer.WriteEndElement("Data");
}

Again no guarantees that this exact code will work as-is (or that it even compiles), but I think that is the idea you're looking for. Stream data from File1 first and write it directly out to the result file. Then, stream data from File2 and write it out. At no point should a full XML file be in memory.

Answer 4

If you run on 64bit, try this: go to your project properties -> build tab -> Platform target: change "Any CPU" to "x64".

This solved my problem for loading huge XML files in memory.

Answer 5

you have to go to file system, unless you have lots of RAM one simple approach:

File.WriteAllText("output.xml", "<Data>");
File.AppendAllText("output.xml", File.ReadAllText("xml1.xml"));
File.AppendAllText("output.xml", File.ReadAllText("xml2.xml"));
File.AppendAllText("output.xml", "</Data>");

another:

var fNames = new[] { "xml1.xml", "xml2.xml" };
string line;
using (var writer = new StreamWriter("output.xml"))
{
    writer.WriteLine("<Data>");
    foreach (var fName in fNames)
    {
        using (var file = new System.IO.StreamReader(fName))
        {
            while ((line = file.ReadLine()) != null)
            {
                writer.WriteLine(line);
            }
        }
    }
    writer.WriteLine("</Data>");
}

All of this with the premise that there is not schema, or tags inside xml1.xml and xml2.xml If that is the case, just code to omit them.