3

I have to deliver an application as a standalone Matlab executable to a client. The code include a series of calls to a function that internally creates several cell arrays.

My problem is that an out-of-memory error happens when the number of calls to this function increases in response to the increase in the user load. I guess this is low-level memory fragmentation as the workspace variables are independent from the number of loops.

As mentioned here, quitting and restarting Matlab is the only solution for this type of out-of-memory errors at the moment.

My question is that how I can implement such a mechanism in a standalone application to save data, quit and restart itself in the case of out-of-memory error (or when high likelihood of such an error is predicted somehow).

Is there any best practice available?

Thanks.

Community
  • 1
  • 1
imriss
  • 1,815
  • 4
  • 31
  • 46

2 Answers2

2

This is a bit of a tough one. Instead of looking to restart to clear things out, could you change the code to break the work in to chunks to make it more efficient? Fragmentation is mostly proportional to the peak cell-related memory usage and how much the size of data items varies, and less to the total usage over time. If you can break a large piece of work in to smaller pieces done in sequence, this can lower the "high water mark" of your fragmented memory usage. You can also save on memory usage by using "flyweight" data structures that share their backing data values, or sometimes converting to cell-based structures to reference objects or numeric codes. Can you share an example of your code and data structure with us?

In theory, you could get a clean slate by saving your workspace and relevant state out to a mat file and having the executable launch another instance of itself with an option to reload that state and proceed, and then having the original executable exit. But that's going to be pretty ugly in terms of user experience and your ability to debug it.

Another option would be to offload the high-fragmentation code in to another worker process which could be killed and restarted, while the main executable process survives. If you have the Parallel Computation Toolbox, which can now be compiled in to standalone Matlab executables, this would be pretty straightforward: open a worker pool of one or two workers, and run the fraggy code inside them using synchronous calls, periodically killing the workers and bringing up new ones. The workers are independent processes which start out with non-fragmented memory spaces. If you don't have PCT, you could roll your own by compiling your application as two separate apps - the driver app and worker app - and have the main app spin up a worker and control it via IPC, passing your data back and forth as MAT files or bytestreams. That's not going to be a lot of fun to code, though.

Perhaps you could also push some of the fraggy code down in to the Java layer, which handles cell-like data structures more gracefully.

Changing the code to be less fraggy in the first place is probably the simpler and easier approach, and results in a less complicated application design. In my experience it's often possible. If you share some code and data structure details, maybe we can help.

Andrew Janke
  • 23,508
  • 5
  • 56
  • 85
  • 1
    +1 good answer as always. FWIW, here are the two exceptions thrown that are associated with out-of-memory errors (identifier and message): `MException('MATLAB:nomem','Out of memory. Type HELP MEMORY for your options.')` and `MException('MATLAB:pmaxsize','Maximum variable size allowed by the program is exceeded.')`. Perhaps one can trap such errors and warn the user that they might need to restart MATLAB – Amro Apr 24 '13 at 20:57
  • Thanks. And good point. If it only happens occasionally you could get away with that and could be less work. But might be a bit more work than it appears, though, because in a GUI, those errors will typically happen inside a callback function, and AFAIK there's no single "top level" event loop or context where you could put the try/catch for it. Every HG callback invocation is effectively its own top level stack frame. You'd have to write the logic in to every callback function, or automagically wrap every callback in an error handler. (Which is doable; I've seen a coworker do it. But tricky.) – Andrew Janke Apr 24 '13 at 23:53
  • hmm you're right, this is not as easy as wrapping the whole code in a try/catch block :\ – Amro Apr 25 '13 at 13:33
  • Thanks Andrew. For code sample, I have to clean it up because there are many calls involved. I will post the sample code soon. For the third solution, to offload that function to another app, I have a question: Do these two apps share the same MCR instance? – imriss Apr 25 '13 at 13:59
  • No, they don't share an MCR instance ("session" is the term Matlab uses). They will link against the same MCR libraries, and maybe against your same deployed code, but each compiled Matlab process gets its own independent MCR session with its own memory space and execution threads. They're isolated so they won't frag each other's memory, and if you want to get fancy, you can run them concurrently to maybe speed up your overall execution. – Andrew Janke Apr 25 '13 at 14:13
2

Another option is to periodically check for memory fragmentation with a function like chkmem.

You could integrate this function to be called silently from you code each couple of iterations, or use a timer object to have it called every X minutes...

The idea is to use thse undocumented functions feature memstats and feature dumpmem to get the largest free memory blocks available in addition to the largest variables currently allocated. Using that you could make a guess if there is a sign of memory fragmentation.

When detected, you would warn the user and instruct them you how to save their current session (export to MAT-file), restart the app, and restore the session upon restart.

Amro
  • 123,847
  • 25
  • 243
  • 454
  • I like this; more feasible than wrapping all callbacks. You don't need to use `feature memstats` or `dumpmem` any more. The documented `memory` function now provides programmatic, structured access to the same data. Much nicer to work with. You could also look at `lasterror` to see if an out-of-memory error had been triggered somewhere else; that's global across call stacks. – Andrew Janke Apr 25 '13 at 13:43
  • Wait, my bad - `memory` only lists the single largest free block; you still need `feature memstats` to get the top 10 or so, which is a useful indicator. Sorry; could have sworn there was another argout with the block list now. – Andrew Janke Apr 25 '13 at 13:52
  • @AndrewJanke: yes of course, no need for undocumented stuff :) Although you could do extra checks with `dumpmem` like seeing where DLLs are loaded in the virtual address space, since you could sometimes benefit from rebasing those offending DLLs that show up in the middle of large contiguous memory blocks – Amro Apr 25 '13 at 13:55
  • Thanks Amro. Actually, there is no UI and interaction with user during the execution. The number of loops is automatically determined as a parameter at the beginning based on the load to be processed. I will update the question to reflect this. – imriss Apr 25 '13 at 14:02
  • @Amro: Guess I would have noticed that if I had read through the chkmem you linked before I went off commenting. That totally makes sense. – Andrew Janke Apr 25 '13 at 14:07
  • @Amro: The idea of rebasing DLLs looks very interesting. Is there any link on how to do it? Thanks. – imriss Apr 25 '13 at 14:16
  • @imriss: Note that if you are deploying applications to customers, I would only offer rebasing DLLs as a last resort option; It's not something a regular user should be doing :) That being said, I found these resources online: [Rebasing DLLs on Windows](http://docs.oracle.com/cd/E19159-01/819-3681/abeio/index.html) and [How do I increase the largest contiguous free memory block available in MATLAB on a 32-bit Win XP SP2?](http://www.mathworks.com/support/solutions/en/data/1-1HE4G5/?solution=1-1HE4G5). I first learned about it from using Cygwin – Amro Apr 25 '13 at 14:44
  • here are some more interesting discussions on memory fragmentation that might be of interest (btw thanks for the suggested edit): http://stackoverflow.com/questions/171205/java-maximum-memory-on-windows-xp, http://stackoverflow.com/questions/103622/tools-to-view-solve-windows-xp-memory-fragmentation – Amro Apr 25 '13 at 14:57
  • @Amro: You are right. However, I am thinking to use it on my workstation because I get out-of-memory error quiet frequently. Thanks. I want to set both yours and Andrew's answer as accepted, but it seems that the system does not allow it. – imriss Apr 25 '13 at 15:12