3

I have a MATLAB script which does complex transforms and calculations in very large 3D matrices (e.g. 50,000 x 300 x 100). I cannot do anything else on the computer while the script is running because the script is very memory-intensive, which is fine; but why does the computer remain slow after the script has finished running and the variables are no longer in the workspace? I assume this must be something to do with the way memory is allocated in the MATLAB script?

[I am running Mac OSX 10.7 and MATLAB 7.13]

user2524828
  • 181
  • 1
  • 4
  • 15
  • 3
    Probably because everything else is swapped out to disk while your script is running, and has to be swapped back into the RAM. Check your hard-drive activity. – Some programmer dude Jan 15 '14 at 11:43
  • Is there a way to minimize the slowness? If it is that everything is being swapped back into RAM, then should this take up to half an hour? There aren't that many background applications running – user2524828 Jan 15 '14 at 11:44
  • I would suggest closing and re-opening MATLAB once the script has finished. – am304 Jan 15 '14 at 12:11
  • Or avoid having to store the 50,000×300×100 array explicitly in memory...Can you give us some more details on the calculation you're doing? Perhaps we can reduce your memory footprint. – Rody Oldenhuis Jan 15 '14 at 12:29
  • @RodyOldenhuis - I start a with a matrix of around 50,000 x 300 and have written a large number of custom functions (~30) to do various matrix manipulations, e.g. deriving another dimension (frequency) from the input matrix / computing circular shifts / algebra in various dimensions. I've spent around 2 months making the computations as efficient as possible, e.g. using logical indexing for circular shifts / compressing data into multi-dimensional matrices to avoid use of for-loops etc. I managed to speed up the runtime by about 20-fold, but perhaps this is at the expense of exhausting memory? – user2524828 Jan 15 '14 at 13:22
  • 2
    @user2524828: As a rule of thumbs, most increases of performance comes at the expense of increasing the memory footprint. Perhaps you can use an approach like [`blockproc`](http://www.mathworks.nl/help/images/ref/blockproc.html)? I.e., do all operations on smaller sections of your giant array? That way you have control over the maximum memory used by the algorithm. Possibly quite useful when the same algorithm is to be re-used on much larger arrays... – Rody Oldenhuis Jan 15 '14 at 13:46
  • Well, that big array takes up just over 11.4 GB of RAM (assuming it's double precision and that there's just one - Matlab may create a copy internally if you weren't careful). How much RAM does your machine have? RAM is cheap so you might get some more regardless. – horchler Jan 15 '14 at 16:07
  • @horchler - I have 8GB of RAM on my computer but I compress out zeros first before loading it, so the matrix that is stored is probably around 50,000x300x10 - still quite large in memory terms I guess. The main issue is if the code goes open-source, people will want to run it on machines with less RAM. I like the block processing idea that RodyOldenhius suggested - will give that a go and see if it improves things. How should I make sure Matlab isn't creating an internal copy? – user2524828 Jan 15 '14 at 21:31
  • It depends on how you're passing around the big array and if and how you're making changes to it. Since you're on a Mac, open up Activity monitor in the Utilities folder and look at how the memory use changes. You can also use `top -u` via the command line (Terminal.app) - [see here](http://support.apple.com/kb/TA20517?viewlocale=en_US&locale=en_US). Come to think of it, the [`purge`](http://osxdaily.com/2012/04/24/free-up-inactive-memory-in-mac-os-x-with-purge-command/) (probably `sudo purge` will be required) command from the Terminal command line might help your slowness issue. – horchler Jan 15 '14 at 22:30
  • 1
    @horchler Thanks for your suggestion. I've been trying to free up active memory to increase the amount available for MATLAB, and concluded: 1) Firefox uses up an obscene amount of memory, 2) MATLAB uses significantly less memory by running it without a display (i.e. 0.8 GB with display, 0.1 without). I started using activity monitor as you suggested, but have moved to piping outputs from top to inform my algorithm and then block process as RodyOldenhuis suggested. Will let you know the outcome! Thanks again. – user2524828 Jan 21 '14 at 10:04
  • Interesting. Yes, Java eats a lot of memory. I don't know how complicated parsing the top call is, but if you just need the percent of memory usage, you could try `ps -o pmem ###`, where `###` is the pid of the Matlab process. This may also be faster too. You will need to get the pid one time at the beginning of your code – [here are several ways](http://stackoverflow.com/questions/11546765/how-to-get-the-pid-of-a-process-by-giving-the-process-name-in-mac-os-x). – horchler Jan 21 '14 at 15:34
  • cool thanks, I've been using top -l 1 | grep -e PhysMem to get the free memory and then working out how much memory would be needed by different sized matrices (the input matrices can be of varying lengths) - this can then inform block processing... your suggestion seems like a good way of looking at the memory footprint during test runs - thanks! – user2524828 Jan 21 '14 at 22:11

1 Answers1

1

You are right, this is due to memory allocation. Matlab requests memory when it is needed but keeps this memory until it closes. You shuld notice that your systems starts working great once you close Matlab. Unfortunately there seems to be no way to onstruct matlab to return the memory to the system.

Hugo
  • 159
  • 2
  • 11
  • 2
    Matlab's`pack` command can be used to recover allocated but unusable memory space. The documentation for `pack` states: 'The pack function does not affect the amount of memory allocated to the MATLAB process. You must quit MATLAB to free up this memory.' – Max Mar 26 '14 at 13:17