I have a list that contain several large files. All the files have the same column names. I want to combine them into an rds file and save with the filename in a column. I have read some of the SO answers and have written the following code. But I ham getting error. This is my attempt:
list.nam<- list.files(pattern="*.I S")
list.fil <- lapply (list.nam, readRDS)
#listfil is 3.8 GB
# running this line gives error
for (i in 1:length(list.fil))
{list.fil[[i]]<-cbind(list.fil[[i]],list.nam[i])
}
data_rbind <- do.call("rbind", list.fil)
colnames(data_rbind)[c(1,2,3,4)]<-c("Time","MRP", "Lot", "Name")
saveRDS(data_rbind, "master.rds")
This is the error:
Error: cannot allocate vector of size 3.9 Mb
In addition: There were 22 warnings (use warnings() to see them)
> sessionInfo()
R version 3.2.2 (2015-08-14)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 8 x64 (build 9200)
locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C
[5] LC_TIME=English_United States.1252
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] tools_3.2.2
Please help.Is the problem I am facing is virtual address space limitation?
Is it possible to do the above operations on-disk using a package like ff? Any help is highly appreciated?
