Reading large XML documents in .net

Question

I need to read large xml using .net files which can easily be several GB of size.

I tried to use XDocument, but it just throws an System.OutOfMemoryException when I try to load the document.

What is the most performant way to read XML files of large size?

If you run on 64bit and want a quick fix for the OutOfMemoryException: go to your project properties -> build tab -> Platform target: change Any CPU to "x64". — BTC, Aug 26 '15 at 09:13

score 13 · Accepted Answer · answered Oct 06 '11 at 08:40

You basically have to use the "pull" model here - XmlReader and friends. That will allow you to stream the document rather than loading it all into memory in one go.

Note that if you know that you're at the start of a "small enough" element, you can create an XElement from an XmlReader, deal with that using the glory of LINQ to XML, and then move onto the next element.

score 5 · Answer 2 · answered Oct 06 '11 at 08:50

The following page makes an interesting read, providing a means to mine data from XML file without loading it in memory. It allows you to combine the speed of XmlReader with the flexibility of Linq:

http://msdn.microsoft.com/en-us/library/bb387035.aspx

And quite an interesting article based on this technique:

http://blogs.msdn.com/b/xmlteam/archive/2007/03/24/streaming-with-linq-to-xml-part-2.aspx

score 0 · Answer 3 · answered Oct 06 '11 at 08:45

0

You could try using an XmlTextReader instance.

http://msdn.microsoft.com/en-us/library/system.xml.xmltextreader.aspx

answered Oct 06 '11 at 08:45

Daniel Becroft

716
3
19

Reading large XML documents in .net

3 Answers3

Linked