Memory usage is growing while processing files

Question

I'm developing an application that reads a xml file folder , and each file it to do some checks and copied to a new folder based on some criteria.

But memory usage continues to grow when it arrives in the foreach loop , and I believe that should not happen , because the variables do not increase at each iteration , are only overwritten.

Here is my code:

using System;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using System.Windows.Forms;
using System.Xml;

namespace XMLOrganizer
{
    public partial class Form1 : Form
    {

        string selectedFolder;

        public Form1()
        {
            InitializeComponent();
            comboBox1.DropDownStyle = ComboBoxStyle.DropDownList;
            comboBox1.SelectedIndex = 0;
        }

        private void button1_Click(object sender, EventArgs e)
        {
            folderBrowserDialog1.ShowDialog();
            selectedFolder = folderBrowserDialog1.SelectedPath;
            organizeBtn.Enabled = true;
        }

        private void organizeBtn_Click(object sender, EventArgs e)
        {

            if (comboBox1.SelectedIndex == -1)
            {
                MessageBox.Show("Selecione o tipo de nota", "Erro!", MessageBoxButtons.OK, MessageBoxIcon.Exclamation,
                    MessageBoxDefaultButton.Button1);
                return;
            }

            if (comboBox1.SelectedIndex != 2)
            {
                OrganizeXml(label2, selectedFolder, comboBox1);
            }

            //ORGANIZAR LOTES
            else
            {
                string folder = selectedFolder;
                label2.Text = "Arquivos sendo processados, aguarde...";
                label2.Refresh();

                string[] files = Directory.GetFiles(folder, "*.xml", SearchOption.AllDirectories);

                int atualFile = 1, totalXML = files.Length;

                foreach (string file in files)
                {
                    XmlDocument xmlDocument = new XmlDocument();
                    xmlDocument.Load(file);
                    XmlNodeList enviNFe = xmlDocument.GetElementsByTagName("enviNFe");
                    string versao = ((XmlElement)enviNFe[0]).Attributes["versao"].Value;
                    XmlNodeList NFe = ((XmlElement)enviNFe[0]).GetElementsByTagName("NFe");

                    Directory.CreateDirectory(selectedFolder + @"\NOTAS");

                    label2.Text = "Processando arquivo " + atualFile + " de " + totalXML;

                    string notaXML;
                    foreach (XmlElement nota in NFe)
                    {
                        notaXML = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><nfeProc versao=\"" + versao + "\" xmlns=\"http://www.portalfiscal.inf.br/nfe\">" + nota.OuterXml + "</nfeProc>";
                        XmlNodeList infNFe = nota.GetElementsByTagName("infNFe");
                        string chave = infNFe[0].Attributes["Id"].Value.Replace("NFe", "");
                        File.WriteAllText(selectedFolder + "\\NOTAS\\" + chave + ".xml", notaXML);
                    }



                }

                OrganizeXml(label2, selectedFolder + "\\NOTAS", comboBox1);



            }
        }

        private static void OrganizeXml(Label label2, string selectedFolder, ComboBox comboBox1)
        {
            string folderMove = String.Empty;

            string folder = selectedFolder;

            label2.Text = "Arquivos sendo processados, aguarde...";
            label2.Refresh();

            string[] files = Directory.GetFiles(folder, "*.xml", SearchOption.AllDirectories);

            int i = 1, arquivos = files.Length;


            Directory.CreateDirectory(folder + @"\ORGANIZADO");
            if (comboBox1.SelectedIndex != 2)
            {
                Directory.CreateDirectory(folder + @"\ORGANIZADO\OUTROS");
                Directory.CreateDirectory(folder + @"\ORGANIZADO\LOTES");
            }
            foreach (string file in files)
            {
                XmlDocument xmlDocument = new XmlDocument();

                try
                {
                    xmlDocument.Load(file);
                    if (xmlDocument.DocumentElement.Name != "nfeProc")
                    {
                        XmlNodeList NFe = xmlDocument.GetElementsByTagName("NFe");
                        var nota = ((XmlElement) NFe[0]);
                        if (nota != null)
                        {
                            XmlNodeList infNFe = ((XmlElement) NFe[0]).GetElementsByTagName("infNFe");
                            string chave = infNFe[0].Attributes["Id"].Value.Replace("NFe", "");
                            string versao = infNFe[0].Attributes["versao"].Value;
                            string notaXML = "<?xml version=\"1.0\" encoding=\"UTF-8\"?><nfeProc versao=\"" + versao +
                                             "\" xmlns=\"http://www.portalfiscal.inf.br/nfe\">" + nota.OuterXml +
                                             "</nfeProc>";
                            string dirNote = Path.GetDirectoryName(file);
                            File.WriteAllText(dirNote + "\\fix_" + chave + ".xml", notaXML);
                        }
                    }



                    //
                    //
                    //

                }
                catch (XmlException)
                {
                    XmlDocument doc = new XmlDocument();
                    string arquivo = ReadFileToString(file);
                    arquivo = RemoveSpecialCharacters(arquivo);
                    if (arquivo == "")
                    {
                        File.Move(file, folder + @"\ORGANIZADO\OUTROS\corrupt_" + Path.GetFileName(file));
                        continue;
                    }
                    try
                    {
                        doc.LoadXml(arquivo);
                        doc.PreserveWhitespace = true;
                        doc.Save(file);
                    }
                    catch (XmlException)
                    {
                        File.Move(file, folder + @"\ORGANIZADO\OUTROS\corrupt_" + Path.GetFileName(file));
                        files = files.Where(f => f != file).ToArray();
                    }
                }
            }

            foreach (string file in files)
            {

                string arquivoLoad = file;
                XmlDocument xmlDocument = new XmlDocument();
                xmlDocument.Load(arquivoLoad);
                XmlNodeList NFe = xmlDocument.GetElementsByTagName("NFe");
                XmlNodeList enviNFe = xmlDocument.GetElementsByTagName("enviNFe");



                if (NFe.Count == 0)
                {
                    if (File.Exists(folder + @"\ORGANIZADO\OUTROS\no_NFe_" + Path.GetFileName(arquivoLoad)))
                    {
                        Random rnd = new Random();
                        File.Copy(arquivoLoad,
                            folder + @"\ORGANIZADO\OUTROS\no_NFe_" + rnd.Next(1, 5000) + Path.GetFileName(arquivoLoad));
                    }
                    else
                    {
                        File.Copy(arquivoLoad, folder + @"\ORGANIZADO\OUTROS\no_NFe_" + Path.GetFileName(arquivoLoad));
                    }

                    continue;
                }

                XmlNodeList infNFe = ((XmlElement)NFe[0]).GetElementsByTagName("infNFe");
                string chave = infNFe[0].Attributes["Id"].Value.Replace("NFe", "");


                if (xmlDocument.DocumentElement.Name != "nfeProc")
                {
                    File.Move(arquivoLoad, folder + @"\ORGANIZADO\OUTROS\no_nfeProc_" + Path.GetFileName(arquivoLoad));
                    arquivoLoad = Path.GetDirectoryName(file) + "\\fix_" + chave + ".xml";
                }
                if (enviNFe.Count > 0)
                {
                    if (File.Exists(folder + @"\ORGANIZADO\LOTES\" + Path.GetFileName(arquivoLoad)))
                    {
                        Random rnd = new Random();
                        File.Copy(arquivoLoad, folder + @"\ORGANIZADO\LOTES\" + rnd.Next(1, 5000) + Path.GetFileName(arquivoLoad));
                    }
                    else
                    {
                        File.Copy(arquivoLoad, folder + @"\ORGANIZADO\LOTES\" + Path.GetFileName(arquivoLoad));
                    }

                    continue;
                }

                //XmlNodeList infNFe = ((XmlElement)NFe[0]).GetElementsByTagName("infNFe");
                XmlNodeList ide = ((XmlElement)infNFe[0]).GetElementsByTagName("ide");


                string tpNF = ((XmlElement)ide[0]).GetElementsByTagName("tpNF")[0].InnerText;

                //if (tpNF == "0") continue;


                XmlNodeList emit = ((XmlElement)infNFe[0]).GetElementsByTagName("emit");

                string emitInfoCod;

                if (((XmlElement)emit[0]).GetElementsByTagName("CNPJ").Count > 0)
                {
                    emitInfoCod = ((XmlElement)emit[0]).GetElementsByTagName("CNPJ")[0].InnerText;
                }
                else if (((XmlElement)emit[0]).GetElementsByTagName("CPF").Count > 0)
                {
                    emitInfoCod = ((XmlElement)emit[0]).GetElementsByTagName("CPF")[0].InnerText;
                }
                else
                {
                    emitInfoCod = "0";
                }

                string ide_dEmi = (((XmlElement)ide[0]).GetElementsByTagName("dEmi").Count > 0)
                    ? ((XmlElement)ide[0]).GetElementsByTagName("dEmi")[0].InnerText
                    : ((XmlElement)ide[0]).GetElementsByTagName("dhEmi")[0].InnerText;
                string[] data = ide_dEmi.Split('-');
                string folderName = data[0] + "\\" + data[1];
                string organizeStyle = String.Empty;

                if (comboBox1.SelectedIndex == 0 || comboBox1.SelectedIndex == 2)
                {
                    organizeStyle = folder + @"\ORGANIZADO\" + emitInfoCod + @"\" + folderName;
                }
                else
                {
                    organizeStyle = folder + @"\ORGANIZADO\" + folderName + @"\" + emitInfoCod;
                }


                if (!Directory.Exists(organizeStyle))
                {
                    Directory.CreateDirectory(organizeStyle);
                }
                folderMove = organizeStyle + "\\";




                if (!File.Exists(folderMove + chave + ".xml"))
                {
                    File.Copy(arquivoLoad, folderMove + chave + ".xml");
                }


                label2.Text = "Arquivos sendo processados, aguarde... (" + i + " / " + arquivos + ")";
                label2.Refresh();
                i++;


            }


            label2.Text = "Notas organizadas com sucesso!";
            label2.Refresh();
        }

        public static string ReadFileToString(string filePath)
        {
            using (StreamReader streamReader = new StreamReader(filePath))
            {
                string text = streamReader.ReadToEnd();
                streamReader.Close();
                return text;
            }

        }

        public static string RemoveSpecialCharacters(string str)
        {
            return Regex.Replace(str, @"[^\u0000-\u007F]", string.Empty);
        }

        private void exitBtn_Click(object sender, EventArgs e)
        {
            Application.Exit();
        }
    }
}

How can I determine what 's going on?

You have multiple 'foreach' loops, have you tried commenting out all but one foreach loop and see which one is constantly increasing memory usage? Xmldocument puts the entire xml into memory so could it just be that further in the iterations the xml file sizes are larger? — bill, Sep 12 '16 at 18:22
Although someone may help further, I think you really need to go back and start eliminating chunks of code until the memory consumption stops. This kind of "binary chunk" elimination will take you 10 minutes and then you can come back to SO to ask "why does XX leak memory" with a much smaller code sample. — PhillipH, Sep 12 '16 at 18:23
You may also want to read something about memory management in .Net. — Alexei Levenkov, Sep 12 '16 at 18:23
May or may not be related, but you shouldn't reinstance Random() in a foreach loop "Random rnd = new Random();" . Create one instance and do rnd.Next() in the loop. With a lot of xml files who knows what you'll get... — bill, Sep 12 '16 at 18:25
@DDDSoft there is no dispose for XmlDocument, it will be garbage collected eventually since the variable only resides within the loop. — bill, Sep 12 '16 at 18:30
Sounds similar to [Memory Leak with XmlDocument()](http://stackoverflow.com/questions/6878974/memory-leak-with-xmldocument). Honestly I would just ditch `XmlDocument` and switch to [LINQ to XML](https://msdn.microsoft.com/en-us/library/bb387098.aspx), the API is much easier to work with. — dbc, Sep 12 '16 at 21:53

score 0 · Answer 1 · answered Sep 12 '16 at 18:29

0

The value of a variable pointing to a reference type is not the object at all, its just the memory address where the object lives.

So when you do the following:

while (true)
{
    var myVariable = new MyReferenceType();
}

The only memory you are really reusing is the variable itself (think of a 32 or 64 bit pointer). But on every iteration your are allocating somewhere in memory space enough to fit the new object you've just created and that memory is most definitely not the memory reserved to the previous object.

This is essentially why your memory usage is growing. The "old" objects of previous iterations with no live reference will eventually get collected by the GC, but that could be never if the GC decides that it has enough memory available to avoid it.

answered Sep 12 '16 at 18:29

InBetween

32,319
3
50
90

so I need to stop using var to instantiate a variable ? – Bruno Henri Sep 12 '16 at 18:33
or for IDisposable classes better use `using()` – Stef Geysels Sep 12 '16 at 18:33
Of course not. Using var has nothing to do with the issue. – InBetween Sep 12 '16 at 18:34
But how can I clean the memory stored for reference types each iteration ? – Bruno Henri Sep 12 '16 at 18:36
@BrunoHenri To put it simply, you can't. You are in a managed environment, there is no deterministic way of reclaiming memory. `IDisposable` objects will deterministically release unmanaged resources (if implemented correctly) but that's about it; in general the managed memory will be reclaimed when the GC decides to. – InBetween Sep 12 '16 at 18:39
How much memory usage are we talking here? Throw your entire code on a never ending loop while(True){} – bill Sep 12 '16 at 18:39

Memory usage is growing while processing files

1 Answers1