11

Short version

Is there a way to prevent R from ever using any virtual memory on a unix machine? Whenever it happens it is because I screwed up and I then want to abort the computation.

Longer version

I am working with a big datasets on a powerful computer shared with several other people. Sometimes I set off commands that requires more RAM than is available, which causes R to start swapping and eventually freeze the whole machine. Normally I can solve this by setting a ulimit in my ~/.bashrc

ulimit -m 33554432 -v 33554432  # 32 GB RAM of the total 64 GB

which causes R to throw an error and abort when trying to allocate more memory than is available. However, if I make a misstake of this sort when parallelizing (typically using the snow package) the ulimit has no effect and the machine crashes anyway. I guess that is because snow launches the workers as separate processes that are not run in bash. If I instead try to set the ulimit in my ~/.Rprofile I just get an error:

> system("ulimit -m 33554432 -v 33554432")
ulimit: 1: too many arguments

Could someone help me figure out a way to accomplish this?

Side track

Why can I not set a ulimit of 0 virtual memory in bash?

$ ulimit -m 33554432 -v 0

If I do it quickly shuts down.

Backlin
  • 14,612
  • 2
  • 49
  • 81
  • See [my answer to the linked question](http://stackoverflow.com/a/24475329/946850) for an R package that allows setting an `ulimit` for a running R sesion. – krlmlr Jun 29 '14 at 10:50

1 Answers1

11

When you run system("ulimit") that is executing in a child process. The parent does not inherit the ulimit from the parent. (This is analgous to doing system("cd dir"), or system("export ENV_VAR=foo").

Setting it in the shell from which you launch the environment is the correct way. The limit is not working in the parallel case most likely because it is a per-process limit, not a global system limit.

On Linux you can configure strict(er) overcommit accounting which tries to prevent the kernel from handling out a mmap request that cannot be backed by physical memory.

This is done by tuning the sysctl parameters vm.overcommit_memory and vm.overcommit_ratio. (Google about these.)

This can be an effective way to prevent thrashing situations. But the tradeoff is that you lose the benefit that overcommit provides when things are well-behaved (cramming more/larger processes into memory).

Kaz
  • 55,781
  • 9
  • 100
  • 149
  • Thanks for your quick reply! I had no idea `system` started a child process, but thought it called its parent. The sysctl solution sounds interesting, especially if it can be set to only limit certain users or groups, so I'll look into it. – Backlin Apr 24 '12 at 18:21