0

I am running the script as follows:

library(ff)
library(ffbase)

setwd("D:/My_package/Personal/R/reading")
x<-cbind(rnorm(1:100000000),rnorm(1:100000000),1:100000000)
system.time(write.csv2(x,"test.csv",row.names=FALSE))

#make ffdf object with minimal RAM overheads 
system.time(x <- read.csv2.ffdf(file="test.csv", header=TRUE,         first.rows=1000, next.rows=10000,levels=NULL))

#make increase by 5 of the column#1 of ffdf object 'x' by the chunk approach 
chunk_size<-100
m<-numeric(chunk_size)

#list of chunks
chunks <- chunk(x, length.out=chunk_size)

#FOR loop to increase column#1 by 5 
system.time(
  for(i in seq_along(chunks)){
    x[chunks[[i]],][[1]]<-x[chunks[[i]],][[1]]+5
  }
)
# output of x
print(x)

#clear RAM used
rm(list = ls(all = TRUE))
gc()

#another option to run garbage collector explicitly.
gc(reset=TRUE)

The issue is that I still some RAM unreleased but all objects and functions have been swept away from the current environment.
Moreover, the next run of the script will increase portion of RAM unreleased as if it is cumulative function (by Task manager in Win7 64bit).

However, if I make a non-ffdf object and sweep it away, the output of rm() and gc() will be Ok.
So my guess about RAM unreleased is connected with specifics of ffdf objects and ff package.

So the effective way to clear up RAM is to quit the current R-session and re-run it again. but it is not very convinient.
I have scanned a bunch of posts about memory cleaning up including this one:
Tricks to manage the available memory in an R session
But I have not found the clear explanations of such a situation and effective ways to overcome it (without resetting R-session). I would be very grateful for your comments.

Community
  • 1
  • 1
Dimon D.
  • 438
  • 5
  • 23
  • **You need to make sure your data is not in RAM in the first place**. Already the 4th line puts your data in RAM. If it fills up your RAM, the behaviour of gc will depend on your operating system which is responsible for freeing up the RAM if you removed it. The issue is not ff here, it is you putting data in RAM, make sure you put only the data in RAM that you need to put in RAM. –  Feb 12 '16 at 10:11
  • @ jwijffels: thanks for the answer. You have confirmed my assumptions about OS memory management. I have done several tests and no extra data had been place into memory. Moreover, If PC is left in the evening I can see the released memory next morning. So the matter is OS memory management. – Dimon D. Feb 12 '16 at 13:29

0 Answers0