3

I am using the ffdf package to do some data pre-processing. My work computer has 4 CPU cores and 8 Gb of RAM, and I can handle about 0.2-0.3 billion data points, which is really wonderful.

However, I have another constraint. The large ffdf objects use up my computer's disk space. When I combine 5 tables into one, and add 20 more columns on it, I got a message "no disk space".

So, I want to remove un-used ffdf objects on my disk drive. Can I do it without exiting the R session? (I have read, write and execute access on the folder.)

I have tried everything I know, like unlink(), file.remove(), delete(), close(), finalize(), finalizer(), ffdrop()".

delete_dir <- "d:/ff/t1_pre"
deletephrase <- "t1_pre"
id <- grep(deletephrase, dir(delete_dir))
todelete <- dir(delete_dir, full.names=T)[id]
todelete
# [1] "d:/ff/t1_pre/t1_pre$b_cnt.ff"     "d:/ff/t1_pre/t1_pre$b_rct.ff"     "d:/ff/t1_pre/t1_pre$b_weit.ff"   
# [4] "d:/ff/t1_pre/t1_pre$deal_id.ff"   "d:/ff/t1_pre/t1_pre$m_id.ff"      "d:/ff/t1_pre/t1_pre$saled_qty.ff"

unlink(todelete)  #only delete .rdata and .rprofile

file.remove(todelete)
# [1] FALSE FALSE FALSE FALSE FALSE FALSE
# Warning messages:
# 1: In file.remove(todelete) :
#   cannot remove file 'd:/ff/t1_pre/t1_pre$b_cnt.ff', reason 'Permission denied'
# 2: In file.remove(todelete) :
#   cannot remove file 'd:/ff/t1_pre/t1_pre$b_rct.ff', reason 'Permission denied'
# 3: In file.remove(todelete) :
#   cannot remove file 'd:/ff/t1_pre/t1_pre$b_weit.ff', reason 'Permission denied'
# 4: In file.remove(todelete) :
#   cannot remove file 'd:/ff/t1_pre/t1_pre$deal_id.ff', reason 'Permission denied'
# 5: In file.remove(todelete) :
#   cannot remove file 'd:/ff/t1_pre/t1_pre$m_id.ff', reason 'Permission denied'
# 6: In file.remove(todelete) :
#   cannot remove file 'd:/ff/t1_pre/t1_pre$saled_qty.ff', reason 'Permission denied'

ffdrop("d:/ff/t2_pre/t2_pre$c_cnt.ff")
# $RData
# d:/ff/t2_pre/t2_pre$c_cnt.ff.RData 
#                              FALSE 

# $ffData
# d:/ff/t2_pre/t2_pre$c_cnt.ff.ffData 
#                               FALSE 

# Warning messages:
# 1: In file.remove(imgfile) :
#   cannot remove file 'd:/ff/t2_pre/t2_pre$c_cnt.ff.RData', reason 'No such file or directory'
# 2: In file.remove(zipfile) :
#   cannot remove file 'd:/ff/t2_pre/t2_pre$c_cnt.ff.ffData', reason 'No such file or directory'

finalize(t2_pre)
# $m_id
# [1] FALSE

# $deal_id
# [1] FALSE

close(t2_pre)
# [1] FALSE

finalizer(t2_pre)
# Error in UseMethod("finalizer") : 
#   no applicable method for 'finalizer' applied to an object of class "ffdf"

delete(t2_pre)
# [1] FALSE
# Warning messages:
# 1: In file.remove(attr(physical, "filename")) :
#   cannot remove file 'd:/ff/t2_pre/t2_pre$m_id.ff', reason 'Permission denied'
Cœur
  • 37,241
  • 25
  • 195
  • 267
DeeKei Song
  • 63
  • 9

1 Answers1

1

You first need to close your ff files before you can do file.remove. Something like close(yourffdf) or close(yourffobject) and next file.remove(list.files(yourpath), recursive=TRUE, full.names=TRUE) will do

require(ff)
x <- as.ffdf(iris)
sapply(filename(x), file.remove) ## Will fail
close(x)
sapply(filename(x), file.remove) ## Works
  • Thank you, your an answer. and I appreciate to your answers. It's really helpful to me when I begin ff package. If there was not your comments, I couldn't get a job. And, I will can't work in Data analysis department. ^^. Thanks a lot!!!! – DeeKei Song Apr 10 '15 at 01:14
  • I already excute close(ffdf) and next file.remove(list.files(path),recursive=T, full.names=T). Sometimes, It's not work. I don't know why. Nowdays, I finding it. When I solve the problem, I will write here. – DeeKei Song Apr 10 '15 at 01:27
  • You can only delete files which are not open. If you are deleting files which are open you get a Permission denied message. So make sure you are only deleting files which are closes as in the example above. –  Apr 10 '15 at 08:02