.NET Memory not freeing up even after exiting the function

Question

I have some C# code that reads a huge file, and after some manipulation, sets its reference to null and exits the function, but the memory won't free up.

XmlDocument xmlDoc = new XmlDocument();
xmlDoc.LoadXml(xmlString);
XmlService.ConvertExcelToXML(xmlDoc);
int sdfid = 320;
XmlService.CompareXML(xmlDoc, ref sdfid, pkid);
xmlDoc.RemoveAll();
xmlDoc = null;

The xmlDoc is a very big string, usually around 50 MB. When I exit the function, that memory is permamnently occupied, and I have to restart my service couple of times a day, otherwise it's memory usage reaches 1GB.

I have tried to use GC.Collect, but no use.

Thank you in advance.

Edit

Here is the class declaration for XmlService. It has no variables. All methods are static

 public class XmlService

ConvertExcelToXML function's code

public static bool ConvertExcelToXML(XmlDocument xmlDoc) {
        XmlNamespaceManager nm = new XmlNamespaceManager(xmlDoc.NameTable);
        nm.AddNamespace("z", "urn:schemas-microsoft-com:office:spreadsheet");
        nm.AddNamespace("o", "urn:schemas-microsoft-com:office:office");
        nm.AddNamespace("x", "urn:schemas-microsoft-com:office:excel");
        nm.AddNamespace("ss", "urn:schemas-microsoft-com:office:spreadsheet");
        nm.AddNamespace("html", "http://www.w3.org/TR/REC-html40");
        XmlNodeList rows = xmlDoc.DocumentElement.SelectNodes("//z:Worksheet/z:Table/z:Row", nm);
        if (rows != null && rows.Count > 0)
        {
            XmlNode nodeNames = rows[0];
            XmlNode nodeValues = rows[1];

            XmlNode destRootNode = xmlDoc.CreateNode(XmlNodeType.Element, "ParentNode", null);
            XmlNode fieldNode = null;
            XmlNode dataNode = null;
                for (int i = 0; i < nodeNames.ChildNodes.Count; i++)
                {
                    if (nodeNames.ChildNodes[i].HasChildNodes)
                    {
                        string nodeName = nodeNames.ChildNodes[i].ChildNodes[0].InnerXml;
                        //string nodeValue = nodeValues.ChildNodes[i].ChildNodes[0].InnerXml;
                        string nodeValue = "DataField" + i.ToString();

                        fieldNode = xmlDoc.CreateNode(XmlNodeType.Element, "Field", null);
                        dataNode = xmlDoc.CreateNode(XmlNodeType.Element, "Data", null);
                        dataNode.InnerXml = nodeName;
                        fieldNode.AppendChild(dataNode);
                        destRootNode.AppendChild(fieldNode);

                        fieldNode = xmlDoc.CreateNode(XmlNodeType.Element, "Field", null);
                        dataNode = xmlDoc.CreateNode(XmlNodeType.Element, "Data", null);
                        dataNode.InnerXml = nodeValue;
                        fieldNode.AppendChild(dataNode);
                        destRootNode.AppendChild(fieldNode);
                    }
                }
            xmlDoc.LoadXml("<ParentNode>" + destRootNode.InnerXml + "</ParentNode>");
            return true;
            } 
            return false;

}

and Code for CompareXML

        public static void CompareXML(XmlDocument filexmlDoc, ref int maxSDFID, string PKID)
    {
        FieldsListBO tmpFieldListBO = null;

        ResponseDTO responseDTO = DbService.getConnection();
        DbConnection con = (DbConnection)responseDTO.ReturnedObjects[Constants.CONNECTION_OBJECT];
        DbProviderFactory factory = (DbProviderFactory)responseDTO.ReturnedObjects[Constants.FACTORY_OBJECT];
        DbCommand cmd = factory.CreateCommand();

        cmd.CommandText = "select * from tree_store";
        cmd.Connection = con;
        con.Open();

        DbDataReader dr = cmd.ExecuteReader();
        dr.Read();
        String pXmlizedString = (String)dr["TransactionTree"];
        dr.Dispose();
        cmd.Dispose();
        con.Dispose();
        XmlSerializer xs = new XmlSerializer(typeof(FieldsListBO));
        MemoryStream memoryStream = new MemoryStream(StringToUTF8ByteArray(pXmlizedString));
        XmlTextWriter xmlTextWriter = new XmlTextWriter(memoryStream, Encoding.ASCII);
        tmpFieldListBO = (FieldsListBO)xs.Deserialize(memoryStream);
        memoryStream.Dispose();
        xmlTextWriter.Close();
        if (tmpFieldListBO.FieldsList.Count < 1)
        {
            maxSDFID = 0;
            return;
        }

        FieldsListBO fieldListBO = new FieldsListBO();

        for (int i = 0; i < tmpFieldListBO.FieldsList.Count; i++)
        {
            if (tmpFieldListBO.FieldsList[i]._pkid.Equals(PKID))
            {
                fieldListBO.FieldsList.Add(tmpFieldListBO.FieldsList[i]);
            }
        }

        GetMaxSDFID(filexmlDoc, ref maxSDFID, fieldListBO);
    }

filexmlDoc being passed to GetMaxSDFID is just being transversed node by node no update/delation is done

post the code for XmlService.ConvertExcelToXML and XmlService.CompareXML — Mitch Wheat, May 01 '11 at 07:39
I suspect the problem is with `XmlService` methods - any chance it's unsafe/unmanaged code? Another option is that it's using `static` variables, thus holding it in memory. — Shadow The GPT Wizard, May 01 '11 at 07:40
Are you sure this piece of code is causing the issue? How is the conversion to Excel done? Or what is done in the CompareXML method. — ChristiaanV, May 01 '11 at 07:40
What happens when the memory reaches 1GB? Do you have actual problems with this (such as the process crashing, the machine getting other problems, or is it more that you are worried about the memory footprint? Is this related: [Memory problems in .NET](http://stackoverflow.com/questions/3787153/memory-probems-in-net)? Also, how is `xmlString` populated? — Fredrik Mörk, May 01 '11 at 07:41
What does XmlService do, does it manipulate the xmlDoc? Can you show us the entire function? — Emond, May 01 '11 at 07:42
@Shadow: I suspect you're on the money. XmlService's methods are are **static**... what's the bet the class is **stateful**? Sigh. — corlettk, May 01 '11 at 07:43
How are you measuring the memory use? Even if the temporary memory allocations are freed by the collector that memory will not generally be freed from the process to the OS, but will be available for further allocations within the process (this will even more try for the large object heap). — Richard, May 01 '11 at 07:54
I have added code please have a look and help me solve the problem — Zahid, Oct 05 '11 at 13:29

Conrad Frix · Answer 1 · 2011-05-01T08:09:54.797

I'm sorry to say this, and I could be wrong, but it really looks like you're guessing at the source of the problem.

I say this is because of what you tried to do. Nulling out a local variable and calling GC.Collect in the hopes that this will fix it. Experienced people will tell you this isn't it and won't help.

Guessing is often a good approach (for instance others have guessed XmlService.ConvertExcelToXML might be a problem), but why guess when you don't have to, get a memory dump of the process when your application has consumed a significant amount of memory. You can use ProcDump but there are many other ways to do this.

Install WinDbg. With this tool you can analyze your memory dump with commands like !dumpheap –stat that can tell you exactly where the 1 GB is going.

score 2 · Answer 2 · answered May 01 '11 at 07:41

We'd need to know what XmlService does in order to have more of a clue of what's going on. It could easily be storing the data for some reason (or possibly for no reason other than poor design).

Assuming the method ends shortly after the code you've shown us, you definitely shouldn't need to set the variable to null, and if XmlService isn't holding a reference to the object, you shouldn't need to call RemoveAll either.

Note that the XmlDocument object itself won't be very large, although its object graph may be. xmlString itself may be large though - if you end up with a lot of objects on the large object heap, that may be causing problems for you, as the LOH isn't compacted even after the space is freed. I wouldn't have expected it to behave quite like this though.

Do you have to get the XML into a string variable to start with? Could you stream it (e.g. from a file) instead?

Large Object Heap :) - For objects over 80K in size – Mitch Wheat May 01 '11 at 07:48 — Mitch Wheat, May 01 '11 at 07:48

.NET Memory not freeing up even after exiting the function

2 Answers2

Linked