Speed decrease in subsetting `data.table` when adding a bracket

Question

I recently noticed in some old code that I had been including extra square brackets when subsetting a data.table and performing a function repeatedly (in my case, calculating correlation matrices). So,

# Slow way
rcorr(DT[subgroup][, !'Group', with=F])

# Faster way
rcorr(DT[subgroup, !'Group', with=F])

(The difference being after subgroup). Just out of curiosity, why does this occur? With the extra brackets, does data.table have to perform some extra computations?

I think [this answer](http://stackoverflow.com/a/27520688/559784) should be of some help, in addition to Richard's nice answer. — Arun, Jul 24 '15 at 06:31

score 6 · Accepted Answer · edited Apr 22 '20 at 16:11

6

Here's a simple interpretation:

# Slow way
rcorr(DT[subgroup][, !'Group'])

The second set of brackets is a second operation on DT, meaning that DT[subgroup] creates a new data table from DT, and then [, !'Group'] operates on that data table, creating another new data table. Hence the decline in speed.

# Faster way
rcorr(DT[subgroup, !'Group'])

This way operates only on DT, all in one go.

edited Apr 22 '20 at 16:11

MichaelChirico

33,841
14
113
198

answered Jul 24 '15 at 00:01

Rich Scriven

97,041
11
181
245

Thanks! I figured something like that was going on. – Chris Watson Jul 24 '15 at 00:11
1

@Chris, it works on the same premise that `[` would in base R, if that helps – Rich Scriven Jul 24 '15 at 00:13

Speed decrease in subsetting `data.table` when adding a bracket

1 Answers1