How do I correctly use the env variable for data.tables within a function

Question

Let us take a simple example

data <- data.table::data.table(a = 1:10, b = 2:11)
j <- quote(c("c") := list(a + 1))
data[, j, env = list(j = j)][]
#        a     b     c
#    <int> <int> <num>
# 1:     1     2     2
# 2:     2     3     3
# 3:     3     4     4
# 4:     4     5     5
# 5:     5     6     6

The above works and produces the correct output. However if I place this inside a function I get a very different output.

data <- data.table::data.table(a = 1:5, b = 2:6)
test <- function(data, ...) {
  dots <- eval(substitute(alist(...)))
  j <- call(":=", call("c", names(dots)), unname(dots))
  print(j)
  data[, j, env = list(j = j)][]
}
test(data = data, c = a + 1)
# `:=`(c("c"), list(a + 1))
#        a     b         c
#    <int> <int>    <list>
# 1:     1     2 <call[3]>
# 2:     2     3 <call[3]>
# 3:     3     4 <call[3]>
# 4:     4     5 <call[3]>
# 5:     5     6 <call[3]>

I assume that the c = a + 1 is just not being evaluated in the correct environment (i.e. the data.table itself).

EDIT: I am using data.table 1.14.3

`?substitute2` examples also covers such problem, it is better to use `str` instead of `print` in such cases — jangorecki, Feb 04 '22 at 20:18

Waldi · Answer 1 · 2022-02-03T21:52:07.293

The difference between

j_quote <- quote(c("c") := list(a + 1))

and

j_call <- call(":=", call("c", names(dots)), unname(dots))

is very subtle:

j_call
#`:=`(c("c"), list(a + 1))
j_quote
#`:=`(c("c"), list(a + 1))

all.equal(j_call,j_quote)
[1] TRUE

identical(j_call,j_quote)
[1] FALSE

This difference lies in the list structure:

# j_quote
as.list(j_quote)
[[1]]
`:=`

[[2]]
c("c")

[[3]]
list(a + 1)

# j_call
as.list(j_call)
[[1]]
`:=`

[[2]]
c("c")

[[3]]
[[3]][[1]]
a + 1

Try:

library(data.table)

data <- data.table::data.table(a = 1:5, b = 2:6)
test <- function(data, ...) {
  dots <- eval(substitute(alist(...)))
  j_call <- call(":=", call("c", names(dots)), call("list",unname(dots)[[1]]))
  j_quote <- quote(c("c") := list(a + 1))
  cat("j_quote",deparse(j_quote),"\n")
  cat("j_call ",deparse(j_call),"\n")
  cat("identical:",identical(j_quote,j_call),"\n")
  data[, j, env = list(j = j_call)][]
}
test(data = data, c = a + 1)
#> j_quote `:=`(c("c"), list(a + 1)) 
#> j_call  `:=`(c("c"), list(a + 1)) 
#> identical: TRUE 
#>        a     b     c
#>    <int> <int> <num>
#> 1:     1     2     2
#> 2:     2     3     3
#> 3:     3     4     4
#> 4:     4     5     5
#> 5:     5     6     6

Mark Fairbanks · Accepted Answer · 2022-02-03T16:23:04.950

It's because dots isn't a call, it's a list of calls. So when data.table evaluates j it's trying to insert that list into a new column.

To fix this you need to splice the list of calls into a single call. You can do this in a call to ':='() directly (Option 1 below), but you can also break this into multiple steps that mirrors what you were doing above by converting dots to be a call to list() (Option 2).

library(data.table)

data <- data.table::data.table(a = 1:5, b = 2:6)

# Option 1 - call to ':='
test <- function(data, ...) {
  dots <- eval(substitute(alist(...)))
  j <- bquote(':='(..(dots)), splice = TRUE)
  print(j)
  data[, j, env = list(j = j)][]
}

# # Option 2 - convert dots to a call to a list
# test <- function(data, ...) {
#   dots <- eval(substitute(alist(...)))
#   dots_names <- names(dots)
#   dots <- bquote(list(..(unname(dots))), splice = TRUE)
#   j <- call(":=", dots_names, dots)
#   print(j)
#   data[, j, env = list(j = j)][]
# }

test(data = data, c = a + 1, double_b = b * 2)
#> `:=`(c = a + 1, double_b = b * 2)
#>        a     b     c double_b
#>    <int> <int> <num>    <num>
#> 1:     1     2     2        4
#> 2:     2     3     3        6
#> 3:     3     4     4        8
#> 4:     4     5     5       10
#> 5:     5     6     6       12

Edit: You can also use test2() if you want to be able to edit the same column or use newly made columns.

test2 <- function(data, ...) {
  dots <- eval(substitute(alist(...)))
  dots_names <- names(dots)
  for (i in seq_along(dots)) {
    dot_name <- dots_names[[i]]
    dot <- dots[[i]]
    j <- call(":=", dot_name, dot)
    print(j)
    data[, j, env = list(j = j)]
  }
  data[]
}

One "issue" with this approach is that you cannot modify columns created within the same call. For example `test(data = data, c = b * 3, d = c * 2)` wouldn't work. — nathaneastwood, Feb 03 '22 at 16:07
In that case you can loop through the dots and evaluate them separately. Edit: See my comment above for `test2()` — Mark Fairbanks, Feb 03 '22 at 16:13

How do I correctly use the env variable for data.tables within a function

2 Answers2