2

I have question regarding memory usage in R. I am running a Rcode on our entire database in R in a for loop. However, the code stops at some point saying that it cannot allocate a vector of size 325.7 mb. When I was looking at the task manager I saw that it was using 28gb of RAM on our server.

I am familiar with the gc() function in R but this does not seems to work. E.g. the code stopped working on the 15th iteration, saying that it cannot allocate the vector. However, if I only run the 15th iteration (and nothing else) there is no problem at all. Moreover, for each new iteration I delete my DT which is by far the largest object in my environment.

Code sample:

DT <- data.table() 
items <- as.character(seq(1:10)) 
for (i in items){ 
  DT <- sample(x = 5000,replace = T) 
  write.csv(DT,paste0(i,".csv")) 
  gc() 
  rm(DT) 
}

I have the feeling that this gc function does not work properly in a for loop. Is that correct or are there any other possible issues, i.e. are there reasons why my memory is full after a few iterations?

Dave van Brecht
  • 514
  • 4
  • 16
  • 1
    "I am running a Rcode on our entire database in R in a for loop." Your problem is not `gc`. I'm sure garbage collection is working as intended. Your problem is your code (which you don't show). I use R quite a lot, but hardly ever need a `for` loop. Please provide a [minimal reproducible example](http://stackoverflow.com/a/5963610/1412059). – Roland Aug 03 '15 at 07:06
  • I have a vector of 1,2,3,...,N and for every i, I am opening a RData file, calculate some Basel II formula's and save it again in a different folder. Hence it look something like this for (i in 1:length(banks)) { load() ... do something save() rm(DT) } – Dave van Brecht Aug 03 '15 at 07:12
  • E.g something like this DT <- data.table() items <- as.character(seq(1:10)) for (i in items){ DT <- sample(x = 5000,replace = T) write.csv(DT,paste0(i,".csv")) gc() rm(DT) } – Dave van Brecht Aug 03 '15 at 07:26
  • 1
    Possible duplicate of [R memory management / cannot allocate vector of size n Mb](http://stackoverflow.com/questions/5171593/r-memory-management-cannot-allocate-vector-of-size-n-mb) – Robert Gowland Apr 15 '16 at 20:45

1 Answers1

0

View the memory limit using the command memory.limit() and then expand it using memory.limit(size=XXX)

Note this is just a temporary approach and I think that this urlR memory management / cannot allocate vector of size n Mb gives a much better explanation on how to tackle these.

mytkavish
  • 97
  • 1
  • 6