2

Is it possible to manually allocate memory in an R session? I work on a shared windows server that sometimes runs into memory issues. When using Stata is it possible to allocate memory for that session with: 

set min_memory 50g

There is something similar in R?

Edit to provide better context:

Why would I need such a command?

Let's say we have a script that peaks the RAM usage at 20Gb but only uses 5Gb most of the time. The natural way that R handle this is to allocate the memory only as needed( which in most cases seems reasonable). But imagine you need to run this program on a shared server( with a lot more RAM than that) that can, due to other users needs, not have 20Gb free memory at the point the program peaks.

In that situation I think would be nice to avoid such a risk by previous allocating the 20Gb you will need for that R session( as it is possible in Stata) when starting the program, and then avoiding to waste resources in a run that will not even finish.

Nícolas Pinto
  • 363
  • 2
  • 14
  • 5
    `memory.limit()` – HFBrowning Jul 27 '17 at 19:36
  • @HFBrowning `memory.limit()` returns the maximum allocation( in most cases the total RAM of your computer). I would like to *change* the *minimun* memory ( the amount of memory allocated to the process). It would be something equivalent to creating a big fake object that uses the memory and then deleting it when it is needed( so no other process or user could have used it first). – Nícolas Pinto Jul 27 '17 at 20:05
  • @HFBrowning That would change the maximum amount of memory that the r session can allocate, wich is not what I am trying to do. – Nícolas Pinto Jul 27 '17 at 20:14
  • I don't think I'm really following how what you want to do is different than what `memory.limit()` can accomplish - have you looked at: https://stackoverflow.com/questions/1395229/increasing-or-decreasing-the-memory-available-to-r-processes ? – HFBrowning Jul 27 '17 at 20:21
  • @HFBrowning Yes, I looked into it, thanks for the reference. `memory.limit()` is just a option for the maximum memory that your r session can allocate, it will be the same unless you change it, even if another process is using 99% of the RAM. It could be that the limit is less than the actual memory available so change it will solve the problem but is not my case. I would like to previous allocate an amount of memory so no other process can use it. – Nícolas Pinto Jul 27 '17 at 20:29
  • 1
    @HFBrowning I haved edited the question to provide better context on why would I need such a command, and maybe that clarifies why I think `memory.limit()` do not help – Nícolas Pinto Jul 31 '17 at 19:15
  • Thanks for the clarification, that makes a lot more sense. This is actually a very good question! I'll do a little digging to see what I can find, and maybe someone else knows – HFBrowning Jul 31 '17 at 19:48

1 Answers1

2

TL;DR: I did a lot of digging and could not find anything that even hinted at a "minimum memory size" setting.

If I understand your question correctly, you would like to have R request a very large (20 Gb) amount of memory up front for a session and then manage this memory on its own in order to ensure that resources are available when needed. From what I have been able to gather, this is not possible and it is approximately the opposite of the strategy employed by R’s designers.

As you have stated, R allocates and manages memory by:

  1. Setting a max limit for the session
  2. Allocating as objects are created
  3. Within certain limits, reclaiming memory as soon as the object is done

For small allocations, I think step 3 is done using the C function alloca() which uses the stack and for larger ones it uses malloc() which allocates on the heap. What this effectively means is that R can request a small pool of memory (128 bytes, according to Advanced R) to manage itself for small objects that come and go pretty quickly, but for larger objects it lets the operating system handle it.

All of the focus I have seen from comments in the source and R internals suggests that the dev team was focused on writing software that would only hold the amount of memory needed for current computation and then promptly release it:

We want the heap sizes to get set at levels adequate for the current computations. The present mechanism uses only the size of the current live heap to provide information about the current needs; since the current live heap size can be very volatile, the adjustment mechanism only makes gradual adjustments. A more sophisticated strategy would use more of the live heap history (lines 314 – 320 in source)

That being said, there could be a package or extension that has re-written some of this core functionality that I am not aware of – although this seems like such a dramatic departure from the way R was written it would probably be a fork of the project rather than a package. If you are not interested in doing that yourself, I would recommend a more mundane solution, like running your program overnight to avoid competition for resources. Or incrementally processing your data, or seeing if something like a database or the bigmemory package could help you. Good luck!

HFBrowning
  • 2,196
  • 3
  • 23
  • 42