21

Under a 32-bit operating system, where maximum memory allocated to any one program is limited, Mathematica gracefully terminates the kernel and returns a max memory allocation error.

On a 64-bit OS however, Mathematica will freely use all the memory available and grind the system to a halt. Therefore, what is the correct way to cap memory usage? One could use MemoryConstrained combined with $Pre or CellEvaluationFunction but I would rather not tie up either of those for this purpose, or have to modify existing uses to incorporate this function.

Is there another way to globally restrict memory usage, such as a kernel flag, or system $Option?

Anthony Geoghegan
  • 11,533
  • 5
  • 49
  • 56
Mr.Wizard
  • 24,179
  • 5
  • 44
  • 125
  • 1
    A good operating system should be able to present to the user program a managed view of its resources, including CPU and memory. – Dr. belisarius Oct 21 '11 at 20:53
  • @belisarius speak plainly man, what are you saying? ;-) Is there an option to set maximum RAM under Windows 7? Of course this assumes Windows is good... – Mr.Wizard Oct 21 '11 at 20:57
  • 3
    I was talking about _good_ operating systems ;) ... No, really I don't know about Win7 – Dr. belisarius Oct 21 '11 at 21:01
  • @belisarius I was afraid of that. The Unix crowd always laughs at what Windows doesn't do. I cannot recall, what OS do you use? – Mr.Wizard Oct 21 '11 at 21:16
  • 1
    OS x also allows mma to grind it to a halt, and, from what I recall from when I was using it, Linux does, too (things may have changed since then though). So a good question. – acl Oct 21 '11 at 21:30
  • @belisarius I guess I didn't forget; I don't think I knew that. acl, interesting, thank you. – Mr.Wizard Oct 21 '11 at 21:48
  • 1
    @Mr. I guess it could be done [by using this](http://msdn.microsoft.com/en-us/library/ms684161%28VS.85%29.aspx), but not without some pain – Dr. belisarius Oct 21 '11 at 22:17
  • @belisarius your suggestion is not unwelcome, but I am looking for a *Mathematica* method. Otherwise, I would have posted to SuperUser, asking how to limit the memory use of a generic application. – Mr.Wizard Oct 21 '11 at 22:28
  • @Mr. One should not make a program to behave like that. The operating system provides you with a virtual environment, so you don't really know what reality is. For example, the OS could compress (ziplike) your memory pages without your knowledge nor acknowledge, and your perceived memory usage could be off by zillions. I saw some beasts that implemented things like that, but is not usually a good idea. – Dr. belisarius Oct 21 '11 at 23:10
  • @belisarius I am not sure what you are referring to. Are you suggesting that there should not be a self-limiting option in Mathematica? In that case, it already has one, `MemoryConstrained`, but I am looking for a different way to apply it. If you are saying that the operating system should not set a hard limit on a program, I don't understand; I would rather the OS said "MathKernel has used too much RAM and has been terminated. Goodbye" than to lock up the whole system. I realize that true RAM use is not going to be the same as calculated RAM use in many cases, but I can accept that. – Mr.Wizard Oct 21 '11 at 23:46
  • @Mr. What I am saying is that the memory (real or virtual) a program _thinks_ it is using may differ from what the OS is really assigning to it. The limit should be managed by the OS and not the program. As for the MemoryConstrained thing, I believe the Mma team is trying to compensate OS deficiencies. – Dr. belisarius Oct 22 '11 at 00:02
  • This reminds me of the old Mac OS (sans 'X') where you sometimes had to manually increase the amount of memory a program was allowed to use. Personally I'd use the `$Pre` approach. – Brett Champion Oct 22 '11 at 00:07
  • @Brett Setting memory limits for processes is a serious concern for OS designers http://publib.boulder.ibm.com/infocenter/zos/v1r12/index.jsp?topic=%2Fcom.ibm.zos.r12.bpxb200%2Fstepse.htm - BTW How was the conference? – Dr. belisarius Oct 22 '11 at 00:40
  • The conference was good; got to meet Sjoerd and Leonid in person, unfortunately mostly in passing. And they scheduled my talks (or my teammates') at the same time as other talks I would really have liked to see. – Brett Champion Oct 22 '11 at 01:36
  • @Brett I wish I'd be able to go. Perhaps next time! – Dr. belisarius Oct 22 '11 at 01:54
  • 1
    @Mr. Probably one can use [belisarius' idea](http://msdn.microsoft.com/en-us/library/ms684161%28VS.85%29.aspx) from inside *Mathematica* by using `NETLink`. – Alexey Popkov Oct 22 '11 at 08:42
  • 1
    @Mr. Wizard I posted my attempt to solve this here (the function I called `totalMemoryConstrained`) here: http://stackoverflow.com/questions/6405304/memory-use-of-apply-vs-map-virtual-memory-use-and-lock-ups/6408489#6408489 . I have an impression though that this does not always solve the problem. See if this can work for you. – Leonid Shifrin Oct 22 '11 at 15:57
  • @Alexey, I have almost no experience with MathLink/NETLink. Would you at least give some tips for implementing this in an answer below? Leonid, thank you, I shall read through that question/answer later, and vote accordingly. :-) – Mr.Wizard Oct 22 '11 at 17:59
  • @Mr. I have no much experience with .NET, so I cannot say how to call the `Job` functions. But I can refer you to a basic `NETLink` Documentation page: "[Setting the Kernel Process Priority](http://reference.wolfram.com/mathematica/notebooks/ProcessPriority.nb)." You will see that it is really simple to manipulate the kernel process priority. And it is also easy to get information on the kernel current memory usage. In this way, one can write a simple monitoring program which will check how many memory this process currently uses. But probably the idea with `Job` objects is much better. – Alexey Popkov Oct 22 '11 at 18:20
  • @Mr. You can also be interested in my answer: "[How to kill slave kernel securely?](http://stackoverflow.com/questions/5229294/how-to-kill-slave-kernel-securely/5239371#5239371)" – Alexey Popkov Oct 22 '11 at 18:22
  • @Mr. As an alternative, you can also be interested in my unpublished function which implements `FreeMemoryConstrained` functionality. But it is really very complex and currently works completely correctly only with version 7. I cannot publish it as an answer because it is huge. But I can give some code snippets which demonstrate basic ideas. Please note that it requires running two MathKernel processes: (1) "master" which monitors the current memory usage of (2) "slave" process in which all the target computations are performed. – Alexey Popkov Oct 22 '11 at 18:36
  • @Alexey _Perhaps_ you could use only one kernel and `RunScheduledTask` – Dr. belisarius Oct 22 '11 at 19:35

1 Answers1

14

In Mathematica 8 you could start a memory watchdog, something along the lines of:

maxMemAllowed        = 15449604;
intervalBetweenTests = 1; (*seconds*)
iAmAliveSignal       = 0;
Dynamic[iAmAliveSignal]
RunScheduledTask[
       If[MemoryInUse[] > maxMemAllowed , Quit[], iAmAliveSignal++],      
       intervalBetweenTests];

Remember to run

RemoveScheduledTask[ScheduledTasks[]];

to disable it.

Edit

You may alert or interactively decide what to do before quitting. As requested, here is a trial with 1.3GB allocated. I can't go much further than that in this machine.

maxMemAllowed = 1.3 1024^3; (*1.3 GB*)
intervalBetweenTests = 1; (*Seconds*)
iAmAliveSignal = 0;
leyendToPrint = "";
Dynamic[leyendToPrint]
RunScheduledTask[
  If[MemoryInUse[] > maxMemAllowed, 
   CreateDialog[CancelButton["Max Mem Reached", DialogReturn[]]]; 
   Quit[],
   Print["Memory in use: ", MemoryInUse[]]; 
   leyendToPrint = 
    "Seconds elapsed = " <> ToString[iAmAliveSignal++]], 
  intervalBetweenTests];
IntegerPartitions[320, {15}];

enter image description here

Dr. belisarius
  • 60,527
  • 15
  • 115
  • 190
  • 1
    Edit acknowledged. If we can get confirmation that this works across other operating systems, I will accept the answer. Thank you again. – Mr.Wizard Oct 22 '11 at 22:12
  • @belisarius +1 Interesting idea. Can this function be extended for restarting the kernel with the same or new code if the previous session was not a fresh MathKernel session? – Alexey Popkov Oct 23 '11 at 02:06
  • @Alexey I really don't know :( – Dr. belisarius Oct 23 '11 at 03:04
  • @belisarius I have created separate question on it: "[Self-restarting MathKernel - is it possible in Mathematica?](http://stackoverflow.com/q/7864643/590388)" – Alexey Popkov Oct 23 '11 at 06:08
  • it's ridiculoous that the program itself doesn't have an option to cap the memory usage. 11 versions and still the same problem. – LowFieldTheory Aug 22 '19 at 10:31