3

I am having a problem with a for loop, where the memory usage keeps gradually increasing until there is no RAM left. The for loop goes for about 10,000 iterations and it each it should read an indexed file in a directory, perform some operations using fmincon, and then save the output to another indexed csv file. Each of the files is smaller than 1MB, and after processing 20 files the memory usage goes from 1GB to 2GB. (Initially, I thought the problem was using 'parfor', but I realized that even in a normal for loop I see this memory "leak"; click here for this question ). The code follows below, where 'my_func" is the function I am minimizing. I can provide this function upon request, but since it is "encapsulated" I would think it would not matter.

list = dir('~/Documents/matlab_files/*.csv');
L    = length(list);
for i = 1:L
      data = readtable(strcat('~/Documents/matlab_files/',list(i).name));
      all_para = [0.03,0.3,0.001,0.001];
      try
          [x,fval] = fmincon(@(all_para)my_func(all_para,data),...
              all_para,[],[],[],[],...
              [-5,-5,0.01,0.01],...
              [5,5,5,5]);
          csvwrite(strcat('~/Documents/matlab_files/output/',list(i).name,'.csv'),x);
      catch ME
              %fprintf('without success: %s\n', ME.message);
              %continue;  % Jump to next iteration of: for i
      end
  end

Again, I wanted to use parfor originally, but I realized that the increase in memory usage happens in the regular for loop as well.

I am using 2018b

dleal
  • 2,244
  • 6
  • 27
  • 49
  • Are you seeing the used memory (resident set size) or the assigned memory (virtual size) increase? See this unrelated answer for the difference between these two sizes: https://stackoverflow.com/a/51997815/7328782 – Cris Luengo Oct 08 '18 at 05:20
  • If it is the resident set size that increases, check to see if it stops happening if you remove one of the functions (`fmincon`, `csvwrite`) -- you might have found a bug. If it is the virtual size that increases, you can stop worrying. This is normal and unimportant. – Cris Luengo Oct 08 '18 at 05:22
  • The function is not important, unless you have some `persistent` or `global` variable in it. Its not common, but can you confirm this is not happening? – Ander Biguri Oct 08 '18 at 08:58
  • I wouldn't recommend to place `try/catch` statements _inside_ the loop. To be able to `try` some code, Matlab has to set up a new workspace/address space and you're asking that at each iteration. If the garbage collection isn't as fast as your loops, you may end up with many unreleased ressources after a few iterations. Try to build your code with the loop _inside_ the `try` statement (if necessary check the type of errors you potentially get with `fmincon` and `csvwrite` and try to handle them yourself). – Hoki Oct 08 '18 at 16:42

0 Answers0