How to load XML in a program quickly?

Question

I'm creating a small tool and this tool will include a treeview of XML. There's no problem dealing with XML files with a small file size, but when I try to load a large XML file (21MB in size), my application becomes unresponsive and takes too much time to load the XML, and most of the time it doesn't load the XML at all. Is there any fix or tweak to make the below codes faster?

public TreeView(string filename)
        {
            InitializeComponent();

            this.Text = filename;

            XmlDataDocument xmldoc = new XmlDataDocument();
            XmlNode xmlnode;
            FileStream fs = new FileStream(filename, FileMode.Open, FileAccess.Read);
            xmldoc.Load(fs);
            xmlnode = xmldoc.ChildNodes[1];
            treeView1.Nodes.Clear();
            treeView1.Nodes.Add(new TreeNode(xmldoc.DocumentElement.Name));
            TreeNode tNode;
            tNode = treeView1.Nodes[0];
            AddNode(xmlnode, tNode);
        }

        private void AddNode(XmlNode inXmlNode, TreeNode inTreeNode)
        {
            XmlNode xNode;
            TreeNode tNode;
            XmlNodeList nodeList;
            int i = 0;
            if (inXmlNode.HasChildNodes)
            {
                nodeList = inXmlNode.ChildNodes;
                for (i = 0; i <= nodeList.Count - 1; i++)
                {
                    xNode = inXmlNode.ChildNodes[i];
                    inTreeNode.Nodes.Add(new TreeNode(xNode.Name));
                    tNode = inTreeNode.Nodes[i];
                    AddNode(xNode, tNode);
                }
            }
            else
            {
                inTreeNode.Text = inXmlNode.InnerText.ToString();
            }
        }

Here is a suggestion, debug using the VS2010 profiler and see what part of your code is taking the longest. Also see if you could use some of the Parallel libraries from MSDN. — user959729, Jan 15 '13 at 15:41
I'd love to have that VS2010, but unfortunately where using SAP technologies in our office and I'm just trying to create a tool using an open source IDE — yonan2236, Jan 15 '13 at 15:46

score 2 · Answer 1 · answered Jan 15 '13 at 15:39

You will have to separate the loading part from building the TreeView.

The loading can then happen on a background thread (BackgroundWorker).

Everything related to the TreeView has to happen on the Main thread but you can speed it up by using SuspendLayout et al.

dutzu · Answer 2 · 2013-01-15T15:49:25.460

1

For the unresponsive part, you can load the xml in a BackgroundWorker asynchronously, and perhaps the whole process of building the treeview structure can be done that way and loaded in the control itself when done.

This might not improve speed a lot, but it will keep the UI responsive. Keep in mind that you cannot access the UI from the code run in the BackgroundWorker directly, you have to call it through the Dispatcher like this:

this.Dispatcher.Invoke(DispatcherPriority.Normal, new Action(() =>
{
     // Execute code that works with UI controls
}));

edited Jan 15 '13 at 15:49

answered Jan 15 '13 at 15:39

dutzu

3,883
13
19

it's always an issue if you are running code in a separate thread and you want to access the UI, the rule is that only the thread that created the controls will be able to modify them so there is a built in Dispatcher and you can issue calls to execute code on the UI thread from a background thread – dutzu Jan 15 '13 at 15:46
so which part of my code should be enclosed to background worker? would you be so kind to help me on this? – yonan2236 Jan 15 '13 at 15:48
2

Dispatcher is for WPF, this was tagged WinForms. So look for Control.Invoke – H H Jan 15 '13 at 15:48
Well, first thing is that you are using WinForms and as @HenkHolterman said above, you need to use Control.Invoke rather than Dispatcher.Invoke. – dutzu Jan 15 '13 at 17:40

score 1 · Answer 3 · answered Jan 15 '13 at 15:39

1

You could try and use the Parallel.ForEach for child nodes to split things across cores. This will be tricky to implement though.

http://msdn.microsoft.com/en-us/library/dd991486.aspx

answered Jan 15 '13 at 15:39

fkerrigan

270
2
9

@dutzu - It probably would, since each child node could be processed independently. Basically, you'd call `Parallel.ForEach(AddNode)` from within `AddNode`, and spawn lots and lots of threads... – Bobson Jan 15 '13 at 15:42
But the main datastructure belongs to the GUI. That make Parallel unsuitable. – H H Jan 15 '13 at 15:49

score 0 · Answer 4 · answered Jan 15 '13 at 15:44

Two minor suggestions: Instead of adding a node then looking it up, create it in a temp variable then work with it from there. And you can use a foreach loop instead of a for loop.

Replace

nodeList = inXmlNode.ChildNodes;
for (i = 0; i <= nodeList.Count - 1; i++)
{
    xNode = inXmlNode.ChildNodes[i];
    inTreeNode.Nodes.Add(new TreeNode(xNode.Name));
    tNode = inTreeNode.Nodes[i];
    AddNode(xNode, tNode);
}

with

foreach (var node in inXmlNode.ChildNodes)
{
    tNode = new TreeNode(node.Name);
    inTreeNode.Nodes.Add(tNode);
    AddNode(node, tNode);
}

I doubt it'll make much difference, but it might help a bit.

@yonan2236 - Edited to add a second suggestion. If nothing else, it'll make the code cleaner and easier to read. — Bobson, Jan 15 '13 at 15:49

score 0 · Answer 5 · answered Jan 15 '13 at 15:49

0

Instead of using XmlDataDocument, change the code to use XmlReader as this class offers the highest performance when working with XML files. XmlReader is an abstract class that can read from a file or from any other stream of data. When reading from a file, you don't load the entire document at once.

more information here: Quick overview by shanselman
or here Performance: LINQ to XML vs XmlDocument vs XmlReader

answered Jan 15 '13 at 15:49

RoelF

7,483
5
44
67

Unlikely to help much here. All Xml has to be loaded and it's only 20 MB – H H Jan 15 '13 at 15:51

score 0 · Answer 6 · edited May 23 '17 at 12:17

First off you should really switch to a SAX-like approach when dealing with large files. Refer to What is the best way to parse (big) XML in C# Code? for one possible solution.

Now you have a solid way to read files that won't cause your system to crash no matter how big the file is.

Next step is to fill your treeview, as other as stated you'll need to do this in a separate thread, I'd recommend a backgroundworker as they're very easy to work with.

As for updating the UI when changes happen, I'm not familiar with winforms but if it support ObservableCollection I'd recommend using thous.

How to load XML in a program quickly?

6 Answers6