Read/Write Large XML Files in C#

Question

I am developing an application with XML database. I have large XML files in which I have to read and write data. The problem is I do not want to load the whole XML file in memory also do not want to loop through the whole file because of performance issue. Because if I load the whole file in the memory this will effect the application performance and may crash the application because of memory leek.

I need a sufficient way to write and read the XML into the file which does not effect on performance and memory.

Any help will be appreciated.

Answer 1

If this XML decision is not yours and you have to deal with it (see whole MSDN sample http://msdn.microsoft.com/en-us/library/bb387013.aspx)

static IEnumerable<XElement> StreamCustomerItem(string uri)
{
    using (XmlReader reader = XmlReader.Create(uri))
    {
        XElement name = null;
        XElement item = null;

        reader.MoveToContent();
        while (reader.Read())
        {
            if (reader.NodeType == XmlNodeType.Element)
            {
                while (reader.Read())
                {
                    if (reader.NodeType == XmlNodeType.Element)
                    {
                        name = XElement.ReadFrom(reader) as XElement;
                        break;
                    }
                }

                while (reader.Read())
                {
                    if (reader.NodeType == XmlNodeType.Element)
                    {
                        item = XElement.ReadFrom(reader) as XElement;
                        if (item != null) 
                        {
                            XElement tempRoot = new XElement("Root", new XElement(name));
                            tempRoot.Add(item);
                            yield return item;
                        }
                    }
                }
            }
        }
    }
}

BUT, if you control the decision, PLEASE, you should forget about the XML. There are several options that will help you and your application to work properly without to much hassle.

1. SQL Compact. Nice and easy SQL approach from Microsoft, and does not requiere SQL server instance. http://www.microsoft.com/en-us/sqlserver/editions/2012-editions/compact.aspx
2. SQL Lite. Works with .net and even Windows 8 applications, easy and pretty stable. http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki

You could even use MySQL, MariaDB or anything similar!

Answer 2

Have a look at this, it will give you some idea about fast reading the xml. http://msdn.microsoft.com/en-us/library/system.xml.xmltextreader.aspx

there is some thread about writing xml file in stackoverflow already.

How to write (big) XML to a file in C#?

However, i think if you are looking for really good performance, some database solutions e.g. sqlserver, mongodb might be a better option

Answer 3

Use this link.

Use XmlReader, It is a nice alternative to allowing us to have only the current record into memory which could hugely improve performance.

Edit: Never use Load method.It will load the whole XML file in memory and if this file is quite big, not only the query might take long time to execute but it might fail running out of memory.

Answer 4

Up to certain extent the performance quite depends on .NET version your application is running on. Another quick reference is Microsoft Patterns and Practices article.

There're 4 ways : XMLDocument, XPathNavigator, XmlTextReader, Linq to XML, I think the differences between them are valuable!

XmlDocument:

It represents the contents of an XML file. When loading it from a file, you read the entire file into memory. Generally speaking, XML parsing is much slower if you're using XmlDocument, which is more geared towards loading the whole DOM in the RAM, ... your application's memory consumption might become just like how a caterpillar moves !!

Using the DOM-model and the XmlDocument or XPathDocument classes to parse large XML documents can place significant demands on memory. These demands may severely limit the scalability of server-side Web applications.

XPath or LINQ-To-XML:

If your concern is more on performance I'd personally not recommend using the XPath or LINQ-To-XML Queries. XPathNavigator provides a cursor model for navigating and editing XML data.

XmlReader:

It might help achieving better in performance as compared to XmlDocument. As others have already suggested. XmlReader is an abstract class and provides an API for fast, forward-only, read-only parsing of an XML data stream ... It can read from a file, from an internet location, or from any other stream of data. When reading from a file, you don't load the entire document at once. And that is where it shines on.

XmlTextReader: XmlTextReader, is an implementation of XmlReader. Use XmlTextReader to process XML data quickly in a forward, read-only manner without using validation, XPath, and XSLT services.

The EOL normalization is always on in the XmlReader from XmlReader.Create, which affects the XDocument. The normalization is off by default on the XmlTextReader, which affects the XmlDocument and XmlNodeReader. It can be turned on via the Normalization property.

Design Considerations

• Consider validating large documents
• Use streaming interfaces
• Consider hard-coded transformations
• Consider element and attribute name lengths ( ! )
• Consider using XmlNameTable : https://msdn.microsoft.com/en-us/library/system.xml.xmlnametable%28v=vs.110%29.aspx

Benchmark Test

• https://blogs.msdn.microsoft.com/codejunkie/2008/10/08/xmldocument-vs-xelement-performance/