how to handle large xml files

Question

public XmlNodeList GetNodes(string _logFilePath, string _strXPathQuery)
        {    
                    objXmlDoc = new XmlDocument();
                    objXmlDoc.Load(_logFilePath);
                    XmlNodeList objxmlNodeList = objXmlDoc.SelectNodes(_strXPathQuery);
                    return objxmlNodeList;
        }

<?xml version="1.0" encoding="UTF-8" standalone="true"?>
<AppXmlLogWritter>
<LogData>
<LogID>999992013021110381000001</LogID>
<LogDateTime>20130211103810</LogDateTime>
<LogType>Message</LogType>
<LogFlag>Flag</LogFlag>
<LogApplication>Application</LogApplication>
<LogModule>Module</LogModule>
<LogLocation>Location</LogLocation>
<LogText>Text</LogText>
<LogStackTrace>Stacktrace</LogStackTrace>
</LogData>
</AppXmlLogWritter>

Here Xml file size is 1000MB when i load it into xmlDocument object then gives me an OutOf memory exception.because XMLDocument stores nodes into Memory .I use Xpath query to filter the nodes throughtout xml file.And then bind to listview to dispaly nodes. I read articles regarding how TO HANDLE LARGE XML FILES they told me use XpathQuery. but the problem doesnt solve. what about filestream? or any other idea to load large xml files?*

How many nodes are you expecting in the result? Does it even make sense to show them all in a list view? — Jon Skeet, Feb 11 '13 at 07:00
I believe thats due DOM in your XMLDocument(), I think you are left to use `XmlTextReader(strUrl);` that doesnt put the whole XML into the RAM — Najzero, Feb 11 '13 at 07:02
@Means then having a list of them in memory isn't very practical. What's the concrete problem you're trying to solve? If you tell us we might be able to suggest a different approach. — Spencer Ruport, Feb 11 '13 at 07:07
@Means: 2 million nodes that you need to display, or 2 million nodes in total, only some of which you want to display? Loading 2 million nodes into a list view simply isn't practical - nor would you end up with a good UI. — Jon Skeet, Feb 11 '13 at 07:10
@spencer xml file is used write nodes of error log ..i write those nodes first and then read nodes of error and then display in listview I used pagination to display on listview..2000 records paging — ND's, Feb 11 '13 at 07:12
jon please dont care of listview i have handle using pagination..I am fetching 2000 records each load of listview...Just take care of How to load large xml file — ND's, Feb 11 '13 at 07:14
Writing an error log to a single file isn't very practical either. If you have control over the generation of the file I would rather split files up (for example after a certain file size is reached) than try to read a file of 1 GB. — reinder, Feb 11 '13 at 07:26
How to read chunk of memory from 1 GB or More size XML file..Can u shoe me some code — ND's, Feb 11 '13 at 07:29
Or how to read xml file using xmltextreader and bind to xmlNodelist — ND's, Feb 11 '13 at 07:36
There's plenty of answers on this on stackoverflow already: http://stackoverflow.com/questions/7671958/reading-large-xml-documents-in-net http://stackoverflow.com/questions/482283/in-c-sharp-how-to-display-1gb-xml-data-very-fast-using-xmltextreader http://stackoverflow.com/questions/468948/in-c-sharp-what-is-the-best-way-to-parse-large-xml-size-of-1gb The best thing is to avoid 1GB XML files. — reinder, Feb 11 '13 at 07:36
reinder Just i want to ask using xmlTextReader how i bind node into xmlnodelist because i have to show filter data to client can u show me some code.. — ND's, Feb 11 '13 at 07:46

Darin Dimitrov · Answer 1 · 2013-02-11T08:24:44.207

You could write a method that will use a XmlReader to read the large XML file in chunks.

Start by designing a model that will hold the data:

public class LogData
{
    public string LogID { get; set; }
    public string LogDateTime { get; set; }
    public string LogType { get; set; }
    ...
}

and then a method that will parse the XML file:

public static IEnumerable<LogData> GetLogData(XmlReader reader)
{
    LogData logData = null;
    while (reader.Read())
    {
        if (reader.IsStartElement("LogData"))
        {
            logData = new LogData();
        }
        if (reader.Name == "LogData" && reader.NodeType == XmlNodeType.EndElement)
        {
            yield return logData;
        }
        if (reader.Name == "LogID")
        {
            logData.LogID = reader.ReadElementContentAsString();
        }
        else if (reader.Name == "LogDateTime")
        {
            logData.LogDateTime = reader.ReadElementContentAsString();
        }
        else if (reader.Name == "LogType")
        {
            logData.LogType = reader.ReadElementContentAsString();
        }
        ...
    }
}

Now you could load only the elements you want to display. For example:

using (var reader = XmlReader.Create("someHugeFile.xml"))
{
    IEnumerable<LogData> data = GetLogData(reader).Skip(10).Take(5);
    // Go ahead and bind the data to your UI
}

Another thing you might want to know is how many records do you have in total in your XML file in order to implement pagination effectively. This could be done with another method:

public static int GetTotalLogEntries(XmlReader reader)
{
    var count = 0;
    while (reader.Read())
    {
        if (reader.IsStartElement("LogData"))
        {
            count++;
        }
    }
    return count;
}

thanks for your valueable reply but i didnt understand GetLogData(reader).Skip(10).Take(5); — ND's, Feb 11 '13 at 09:39
That's where you perform the pagination. It's LINQ. You skip 10 records and take 5 more. Basically here you are taking records from 10 to 15. You could easily calculate this in order to implement your pagination. Once you know the current page and the number of records you want per page to be shown as well as the total number of records you could retrieve only the records that need to be shown. — Darin Dimitrov, Feb 11 '13 at 09:40
Simply start iterating over the `IEnumerable` that is returned from the method and discard the first items you are not interested in (that would be the equivalent of the `.Skip` LINQ extension method). Then take as much elements as you need to display (that would be the equivalent of the `.Take` LINQ extension method). — Darin Dimitrov, Feb 11 '13 at 09:52
I have already showed you more than enough code. Now you could try to implement my suggestion and if you encounter some specific difficulties do not hesitate to show your progress so that we could see what might be wrong with it. But writing a simple `for` loop over the resultset returned by the function shouldn't be of an insurmountable difficulty. — Darin Dimitrov, Feb 11 '13 at 09:56
Darin yuor solution is working fine but ine issue I have to filter data..Means uisng xmldoc i used xpath query but what should i do in xmlreader. — ND's, Feb 11 '13 at 14:04

score 0 · Answer 2 · answered Feb 11 '13 at 08:22

I suggest that you compress the xml message before you send them to stream then decompress it once you received it in the other end. In WCF you can do it like this http://www.codeproject.com/Articles/165844/WCF-Client-Server-Application-with-Custom-Authenti#Compression

how to handle large xml files

2 Answers2