Extracting a small subset of data from XMLs

Question

I am writing a C# / VB program that is to be used for reporting data based upon information received in XMLs.

My situation is that I receive many XMLs per month (about 100-200) - Each ranging in size from 10mb to 350mb. For each of these XMLs, I only need a small subset of its data (less than 5% of any one file's entire data) so as to produce the necessary reports.

Also, that subset of data will always be held in the same key-structure (it will exist within multiple keys and at differing levels down, perhaps, but it will always exist within the same key names / the keys containing it will always have the with the same attributes such as "name", etc)

So, my current idea of how to go about doing this is to:

1. To create a "scraper" that will pull the necessary data from the XMLs using XPath.
2. Store that small subset of necessary data in a SQL Server table along with file characteristic data stored in a separate table so as to know which file this scraped data came from
3. Query out the data into a program for reporting it.

My main question here is really what is the best way to scrape that data out? I am most familiar with XPath, but for multiple files of 200MB in size, I'm afraid of performance issues loading in the entire file.

Other things I have seen / researched are:

1. Creating an XSLT file to transform / pull from the XML only the data I want
2. Using Linq to XML
3. Somehow linking the XMLs to SQL server and then being able to query them directly
4. Using ADO to query the XMLs from within the program
5. Doing it using the XMLReader class (rather than loading in each XML entirely)
6. Maybe there is a native .Net component that does this very well already

Quite honestly, I just have no clue what the standard is given the high number of XMLs and the large variance in file sizes and I'm not familiar with any of the other ways of doing this - such as, for example, linking the XMLs to SQL Server directly / using ADO to query the XML - and, therefore, don't know of their possible benefits / drawbacks.

If any of you have been in a similar situation, I'd really appreciate any kind of pointers in the right direction / at least validation that my method isn't the worst one out there :)

Thanks!!!

Answer 1

As for the memory consumption and performance concerns, a nice feature of the .NET XML APIs is that you can combine XmlReader with XPathDocument or XmlDocument or XElement to only selectively read part of a document into memory to then have the XPath or LINQ to XML features available on that part. LINQ to XML has http://msdn.microsoft.com/en-us/library/system.xml.linq.xnode.readfrom%28v=vs.110%29.aspx for doing that, DOM/XmlDocument has http://msdn.microsoft.com/en-us/library/system.xml.xmldocument.readnode%28v=vs.110%29.aspx. So depending on your XML structure you might be able to use an XmlReader to read forward through the XML in a fast way without consuming much memory and then, when you have the element you are interested in, you can read it into an XElement (LINQ to XML) or XmlNode (DOM) to then apply LINQ to XML and/or XPath to read out details.