1

I have a function that makes an assignment by reference to an already-existing data.table. The function works when the data.table is created and input to the function, but it does not work when the dataset is saved to disk, re-loaded and then the same function is run. However, if we do setDT() on the newly loaded data.table (or we make an assignment) and then run the function, then the function works fine. What is going on here? Why do we need to 'reset' the data.table before using this function?

library(data.table)
file.path <- "H:/mydt.RDS"
make_col <- function(dt) { 
  dt[ , z := 1]
}
# this works
mydt <- data.table(a = 1:3)
make_col(mydt)
# but if we save and load the saved copy...
mydt <- data.table(a = 1:3)
saveRDS(mydt, file.path)
# this doesn't work
mydt <- readRDS(file.path)
make_col(mydt)
# but this works
mydt <- readRDS(file.path)
setDT(mydt)
make_col(mydt)
# and so does this
mydt <- readRDS(file.path)
mydt[ , b := 1]
make_col(mydt)
# and so does this 
mydt <- readRDS(file.path)
mydt2 <- copy(mydt)
make_col(mydt2)
Stuart
  • 131
  • 1
  • 4
  • I've had this same issue. Interested to know the reason. – IceCreamToucan Jun 25 '18 at 19:05
  • 1
    @Ryan and OP: type `vignette("datatable-faq")` and search for "Reading data.table from RDS or RData file" – Frank Jun 25 '18 at 19:24
  • Memory needs to be over-allocated (`truelength(mydt) > length(mydt)`) for assignment to new columns by reference. There is nothing to reference if it isn't. `readRDS` doesn't do this over-allocation. – Roland Jun 25 '18 at 19:27
  • Btw, even if your table is not on disk, your function is fragile due to @Roland's point: `DT = data.table(a = 1); DT[, paste0("V", 2:truelength(DT)) := 1]; make_col(DT)` it fails to make a new column `z` in the table you're passing to the function. Check `ncol(DT)` before and after the function and look for `DT$z`. – Frank Jun 25 '18 at 19:31
  • To clarify more: Over-allocating requires that a copy is made. If dt is copied inside the function is is not a reference to mydt anymore and changes to it don't happen to mydt. – Roland Jun 25 '18 at 19:39

0 Answers0