Why the memory is not released after writing files on disk in C#

Question

I have a complicated algorithm which receives data from a socket connections, transformate the data and stores it as soon as possible on the HD..The data, because of that fact I don't want the processing to slow down is stored by using a different thread. The data storage algorithm resembles this structure. It essentially saves XML on the disk.

Begin Thread
 beginthread:

 XmlTextWriter xmltextWriter;
 Save Xml file 1
 xmltextWrite.close();

 XmlTextWriter xmltextWriter;
 Save Xml file 2
 xmltextWrite.close();

 goto beginthread:
End Thread

It works correctly but if I get a look to the Task Manager , I could notice that the amount of memory consumed by my program encreases quickly over the time (500mb after 1 hour of work). This could be justified because of the fact that the thread is not so fast as the data which comes in and the .NET framework stores temporary all in memory for me. But what I didn't understand is why if the incoming socket connections will stop, even after a few minutes that the thread continues to work..the task manager continues to show 500Mb of memory..Why the memory is not relased ?! The XmlTextWriter object is a local variable and is closed every time.

As requested.. This is part of the code

     beginthread:
        if (sleeptime < 1000) sleeptime += 2;

        try
        {

            while (hashBeginConn.Count > 0)
            {
                sleeptime = 0;

                int connToApply = hashBeginConn[0];

                if (olddate.ToShortDateString() != ListsockConnections[connToApply].beginDate.ToShortDateString())
                {
                    JoinDataFromTempFile(ListsockConnections[connToApply].beginDate.Date.Subtract(olddate.Date).Days, false, d);
                    olddate = ListsockConnections[connToApply].beginDate.Date;
                }

                if (tocreate)
                {
                    // XML Serialization
                    XmlTextWriter xmltextWriter;

                    Encoding enc = null;
                    if (ListsockConnections[connToApply].ENCfromCode) enc = Encoding.GetEncoding(ListsockConnections[connToApply].codepage);
                    if (ListsockConnections[connToApply].ENCDefault) enc = Encoding.Default;
                    if (ListsockConnections[connToApply].ENCfromText) enc = Encoding.GetEncoding(ListsockConnections[connToApply].codename);
                    if (enc == null) { enc = null; }

                    // xmltextWriter = new XmlTextWriter(folderPath + "\\" + cacheFileName, enc);
                    xmltextWriter = new XmlTextWriter(DataPath + "\\_temp.xml", enc);
                    xmltextWriter.Formatting = Formatting.Indented;

                    // Start document
                    // xmltextWriter.WriteStartDocument();
                    xmltextWriter.WriteStartElement("ConnectionList");
                    xmltextWriter.WriteStartElement("connection");

                    xmltextWriter.WriteStartElement("ConnectionCounter");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].ConnectionCounter.ToString());
                    xmltextWriter.WriteEndElement();

                    xmltextWriter.WriteStartElement("IDConnection");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].IDConnection.ToString());
                    xmltextWriter.WriteEndElement();

                    xmltextWriter.WriteStartElement("Parsed");
                    xmltextWriter.WriteValue("false");
                    xmltextWriter.WriteEndElement();

                    xmltextWriter.WriteStartElement("connType");
                    xmltextWriter.WriteValue("TCP/IP");
                    xmltextWriter.WriteEndElement();

                    xmltextWriter.WriteStartElement("beginConn");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].beginDate.ToString());
                    xmltextWriter.WriteEndElement();

                    xmltextWriter.WriteStartElement("remoteAddressFamily");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].remoteAdressFamily);
                    xmltextWriter.WriteEndElement();

                    xmltextWriter.WriteStartElement("remoteIP");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].remoteIP);
                    xmltextWriter.WriteEndElement();

                    xmltextWriter.WriteStartElement("localIP");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].localIP);
                    xmltextWriter.WriteEndElement();

                    xmltextWriter.WriteStartElement("remoteport");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].remoteport.ToString());
                    xmltextWriter.WriteEndElement();

                    xmltextWriter.WriteStartElement("localport");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].localport.ToString());
                    xmltextWriter.WriteEndElement();

                    xmltextWriter.WriteStartElement("dataEncoding");
                    if (ListsockConnections[connToApply].codepage != 0 || ListsockConnections[connToApply].codename != "")
                    {
                        if (ListsockConnections[0].codepage != 0)
                        { xmltextWriter.WriteValue(ListsockConnections[connToApply].codepage.ToString()); }
                        else
                        { xmltextWriter.WriteValue(ListsockConnections[connToApply].codename.ToString()); }
                    }
                    else
                    { xmltextWriter.WriteValue("NONE"); }
                    xmltextWriter.WriteEndElement();

                    xmltextWriter.WriteEndElement();
                    xmltextWriter.WriteEndElement();
                    xmltextWriter.Flush();
                    xmltextWriter.Close();

                    tocreate = false;
                }
                else
                {
                    FileInfo fi;
                    FileStream fstream;

                    //fi = new FileInfo(folderPath + "\\" + cacheFileName);
                    fi = new FileInfo(DataPath + "\\_temp.xml");
                    fstream = fi.OpenWrite();

                    XmlTextWriter xmltextWriter;

                    Encoding enc = null;
                    if (ListsockConnections[connToApply].ENCfromCode) enc = Encoding.GetEncoding(ListsockConnections[connToApply].codepage);
                    if (ListsockConnections[connToApply].ENCDefault) enc = Encoding.Default;
                    if (ListsockConnections[connToApply].ENCfromText) enc = Encoding.GetEncoding(ListsockConnections[connToApply].codename);
                    if (enc == null) { enc = null; }

                    xmltextWriter = new XmlTextWriter(fstream, enc);

                    xmltextWriter.Formatting = Formatting.Indented;

                    fstream.Position = fstream.Length - 17;

                    xmltextWriter.WriteRaw("  <connection>" + Environment.NewLine);

                    xmltextWriter.WriteRaw("     <ConnectionCounter>");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].ConnectionCounter.ToString());
                    xmltextWriter.WriteRaw("</ConnectionCounter>" + Environment.NewLine);

                    xmltextWriter.WriteRaw("     <IDConnection>");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].IDConnection.ToString());
                    xmltextWriter.WriteRaw("</IDConnection>" + Environment.NewLine);

                    xmltextWriter.WriteRaw("     <Parsed>");
                    xmltextWriter.WriteValue("false");
                    xmltextWriter.WriteRaw("</Parsed>" + Environment.NewLine);

                    xmltextWriter.WriteRaw("     <connType>");
                    xmltextWriter.WriteValue("TCP/IP");
                    xmltextWriter.WriteRaw("</connType>" + Environment.NewLine);

                    xmltextWriter.WriteRaw("     <beginConn>");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].beginDate.ToString());
                    xmltextWriter.WriteRaw("</beginConn>" + Environment.NewLine);

                    xmltextWriter.WriteRaw("     <remoteAddressFamily>");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].remoteAdressFamily);
                    xmltextWriter.WriteRaw("</remoteAddressFamily>" + Environment.NewLine);

                    xmltextWriter.WriteRaw("     <remoteIP>");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].remoteIP);
                    xmltextWriter.WriteRaw("</remoteIP>" + Environment.NewLine);

                    xmltextWriter.WriteRaw("     <localIP>");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].localIP);
                    xmltextWriter.WriteRaw("</localIP>" + Environment.NewLine);

                    xmltextWriter.WriteRaw("     <remotePort>");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].remoteport.ToString());
                    xmltextWriter.WriteRaw("</remotePort>" + Environment.NewLine);

                    xmltextWriter.WriteRaw("     <localport>");
                    xmltextWriter.WriteValue(ListsockConnections[connToApply].localport.ToString());
                    xmltextWriter.WriteRaw("</localport>" + Environment.NewLine);

                    xmltextWriter.WriteRaw("     <dataEncoding>");
                    if (ListsockConnections[connToApply].codepage != 0 || ListsockConnections[connToApply].codename != "")
                    {
                        if (ListsockConnections[connToApply].codepage != 0)
                        {
                            xmltextWriter.WriteValue(ListsockConnections[connToApply].codepage.ToString());
                        }
                        else
                        {
                            xmltextWriter.WriteValue(ListsockConnections[connToApply].codename.ToString());
                        }
                    }
                    else
                    {
                        xmltextWriter.WriteValue("NONE");
                    }
                    xmltextWriter.WriteRaw("</dataEncoding>" + Environment.NewLine);

                    xmltextWriter.WriteRaw("  </connection>" + Environment.NewLine);
                    xmltextWriter.WriteRaw("</ConnectionList>");

                    xmltextWriter.Flush();
                    xmltextWriter.Close();
                    fstream.Close();


                    if (fi.Length >= (maxFileTempSize * 1000000))
                    {
                        JoinDataFromTempFile(0, false, enc);
                    }
                }

                lock (lockThis)
                {
                    hashBeginConn.RemoveAt(0);
                }

            }

Any chance you can share the actual code or an simple example which mimics the behavior you are seeing? — M.Babcock, Jan 02 '12 at 03:28

score 9 · Accepted Answer · edited May 23 '17 at 12:31

A number of answers say that you must call Dispose. Though those answers mean well, they won't actually help you. You are calling Close, and Close and Dispose do the same thing. It is a better practice to use a "using" block so that you get Dispose called for you automatically, but your code is fine as it is.

The real answer to your question is "stop worrying about it". You are thinking about this at the wrong level. I assume that you are looking at either "working set" or "private bytes" in the task manager, but you might not understand what those actual means. Most people do not. This answer gives a good summary:

What is private bytes, virtual bytes, working set?

OK, now that you know what "private bytes" are, it should be more clear why this is not a problem. Suppose you are the CLR garbage collector. You allocate a bunch of memory on behalf of the user and use it to store managed objects. Every now and then, the garbage collector runs, compacting the objects in memory and marking the memory formerly used by now-dead objects as available. But why would the GC return all those blocks of memory to the operating system? The GC has evidence that you are the sort of person who writes programs who use that much memory, so it keeps the empty pages around so that it does not have to incur the expense of allocating them again later when you use that much memory again two milliseconds from now.

So, stop worrying about it. Everything is probably fine. Using 500 million bytes of memory is not a problem. If you start getting low on virtual memory, the GC will likely start decommitting the unused pages. If it doesn't -- if that keeps growing without bound, then start worrying.

If you're still worried, use the right tool for the job. "Private bytes" tells you very little about what is actually happening with memory in your program. If you want to know what is really happening in the garbage collector then you need to use a managed memory profiler. It will give you a report of exactly what is going on.

Dispose and Close are NOT documented to have the same behavior. See [http://msdn.microsoft.com/en-us/library/system.xml.xmltextwriter.aspx] What evidence do you have that they are identical? — Elroy Flynn, Jan 02 '12 at 23:59
You seem to be stating that memory is never returned to the OS, but by observing private bytes go down you can see that it certainly is. (Though perhaps not all of it, all them time.) In answer to your question "Why would the GC return (the memory)?", the answer is that a well-behaved application should indeed release resources when finished using them. Many applications go through a life cycle that requires more memory at some times than at others. It's not unheard of for an application to require more memory at startup than later. — Elroy Flynn, Jan 03 '12 at 00:20
@ElroyFlynn You're correct, the documentation doesn't indicate that `Dispose()` and `Close()` do the same thing. It so happens that they *do* (`Dispose()` calls `Close()` if the stream isn't already in a closed state) but that's an implementation detail. That being said, it's a detail that is germane to the answer. — dlev, Jan 03 '12 at 01:17
That Dispose() calls Close() doesn't mean that CLose() does everything that Dispose() does. — Elroy Flynn, Jan 03 '12 at 01:59
@ElroyFlynn I should have been more clear: `Dispose()` does nothing else but call `Close()`, so they really are equivalent in this case. — dlev, Jan 03 '12 at 02:13
@ElroyFlynn I used ILSpy to look at the IL. As I said, the docs don't state that this will always be the case, so your point is a valid one. — dlev, Jan 03 '12 at 04:20

score 5 · Answer 2 · edited Jan 02 '12 at 06:07

5

You should be calling xmltextWriter.Dispose().

Also, the memory will not be freed until the garbage collector kicks in. You should normally allow this to happen automatically, but you can explicitly call it using the static method GC.Collect(). It is NOT recommended to call gc.collect explicitly. Let the CLR do it on its own schedule.

edited Jan 02 '12 at 06:07

Mitch Wheat

295,962
43
465
541

answered Jan 02 '12 at 03:39

Elroy Flynn

3,032
1
24
33

to add further detail to what @Elroy said - "on its own schedule" - sometimes the GC doesn't kick in until a certain amount of memory is being used in the application. For mine, I've found that it doesn't do anything until it reaches ~50MB, though it sounds like you're currently well above that. – Origin Jan 02 '12 at 03:46
1

+1: To add: use using blocks around anything that implements IDisposable (and where appropriate, obviously). – Mitch Wheat Jan 02 '12 at 06:07
The OP calls `Close` if no exception occurs, so why should he call the equivalent `Dispose`? `using` is obviously better, but the difference only matters if an exception occurs. And manually calling `Dispose` is certainly no better than manually calling `Close`. – CodesInChaos Jan 02 '12 at 20:25
Dispose and Close are NOT documented to have the same behavior. See [http://msdn.microsoft.com/en-us/library/system.xml.xmltextwriter.aspx] What evidence do you have that they are identical? – Elroy Flynn Jan 02 '12 at 23:59

score 0 · Answer 3 · answered Jan 02 '12 at 06:20

xmltextWriter holds unmanaged resources, in this case a file which is an operating system resource. It won't let go of these resources unless you explicitly call Dispose() on it.

The best practice is to enclose the usage of the writer in a using { } block, so that Dispose() will be called automatically even if an exception is thrown.

Dispose() (or using) does not guarantee that the memory will be freed at that very instant, but it does prevent your application from holding the file open longer than needed.

Why the memory is not released after writing files on disk in C#

3 Answers3