13

I have 6 GB memory in my machine (Windows 7 Pro 64 bit) and in R, I get

> memory.limit()
6141

Of course, when dealing with big data, memory allocation error occurs. So in order to make R to use virtual memory, I use

> memory.limit(50000)

Now, when running my script, I don't have memory allocation error any more, but R hogs all the memory in my computer so I can't use the machine until the script is finished. I wonder if there is a better way to make R manage memory of the machine. I think something it can do is to use virtual memory if it is using physical memory more than user specified. Is there any option like that?

zx8754
  • 52,746
  • 12
  • 114
  • 209
Tae-Sung Shin
  • 20,215
  • 33
  • 138
  • 240

3 Answers3

11

Look at the ff and bigmemory packages. This uses functions that know about R objects to keep them on disk rather than letting the OS (which just knows about chunks of memory, but not what they represent).

Greg Snow
  • 48,497
  • 6
  • 83
  • 110
5

R doesn't manage the memory of the machine. That is the responsibility of the operating system. The only reason memory.size and memory.limit exist on Windows is because (from help("Memory-limits")):

 Under Windows, R imposes limits on the total memory allocation
 available to a single session as the OS provides no way to do so:
 see 'memory.size' and 'memory.limit'.

R objects also have to occupy contiguous space in RAM, so you can run into memory allocation issues with only a few large objects. You could probably be more careful with the number/size of objects you create and avoid using so much memory.

Joshua Ulrich
  • 173,410
  • 32
  • 338
  • 418
4

This is not a solution but a suggestion. Use memory efficient objects wherever possible: for instance, use a matrix instead of a data.frame.

Here an example

m = matrix(rnorm(1000), 2, 2)
d = as.data.frame(m)
object.size(m)
232 bytes
object.size(d)
808 bytes
agstudy
  • 119,832
  • 17
  • 199
  • 261
  • 3
    This example seems to only highlight the very small fixed data allocation difference. Take the matrix to be `m = matrix(rnorm(1000), 20000, 10)` and the size differences between the objects become negligible. In fact checking this for some random `data.frame` I had in my workspace, the `matrix` version was 2 times larger. So I don't think this suggestion is useful for the problem at hand. – eddi Apr 05 '13 at 18:39
  • 1
    No. I disagree. Matrix are more efficient than data.frame. That is one reason why internally, a lot of R functions will coerce to matrices data that are in data frames. I mention clearly that my answer is just a suggestion a best practice if you want. For your second comment ( random data.frame smaller then matrix version) I think it is False. – agstudy Apr 05 '13 at 18:52
  • This is what I see for a specific `data.frame` I have in my workspace. `object.size(as.data.frame(rec)): 11542192 bytes` and `object.size(as.matrix(rec)): 26516160 bytes`. – eddi Apr 05 '13 at 19:25
  • 1
    Here's a reproducible example: `d = data.frame(a = 1:100, b = "b")` - try checking the various object sizes. There is no way to store e.g. numbers and strings together in a `matrix` in a more efficient manner (both for memory and for speed) than in a `data.frame`. Your suggestion will only save a negligible amount of memory (literally about 600 bytes) for uniform type `data.frame`'s and will not work for anything else. – eddi Apr 05 '13 at 19:36
  • @eddi Please try to read [this](http://stackoverflow.com/questions/5158790/data-frame-or-matrix) to get my point about matrix efficiency comparing to a data.frame. – agstudy Apr 05 '13 at 19:50
  • Oh, I get your point, it's just that your point is irrelevant for the problem at hand. Cheers :) edit: note that the OP is about *memory* efficiency – eddi Apr 05 '13 at 20:00