Faster aggregate multiple columns

Question

I have following function which runs 100s of times. This aggregation is the bottleneck in my code. Is it possible to make is faster with just using data.table or rewrite this function using rcpp?

  logit.gr <- function(DT){
    temp1 <- DT[, lapply(.SD, function(x) col1*sum(y*(x - sum(x*exp(col2))))), by = .(main_idx), .SDcols = c('col3','col4')]
    return(-colSums(temp1[, c('col3','col4'), with = F]))
  }

where DT is

DT <- data.table(main_idx = c(rep('A',4), rep('B', 5)), col1 = runif(9), col2 = -2+runif(9), col3 = 1+runif(9), col4 = 1+runif(9), y = runif(9))

Please read the info about [how to ask a good question](http://stackoverflow.com/help/how-to-ask) and how to give a [reproducible example](http://stackoverflow.com/questions/5963269/how-to-make-a-great-r-reproducible-example/5963610). This will make it much easier for others to help you. — Jaap, Apr 09 '18 at 17:09
should it be `col3*sum(y*(x - sum(x*exp(col4)))))` instead of this `col1*sum(y*(x - sum(x*exp(col2)))))` — Sathish, Apr 09 '18 at 18:00
I was calling another function inside this function. I removed it since it was not the bottleneck. Forgot to remove theta. — deepAgrawal, Apr 09 '18 at 18:04
@Sathish for the purpose of making it faster it does not matter which columns do you use. — deepAgrawal, Apr 09 '18 at 18:07

score 1 · Accepted Answer · answered Apr 09 '18 at 18:09

1

I think away to optimize is:

sum should be added in function used in lapply itself. It will result in to only 1 row per main_idx in resultant data.table.

chain of [ operator should be used to sum columns col3 and col4.

library(data.table)
DT[, lapply(.SD, function(x) sum(col1*sum(y*(x - sum(x*exp(col2)))))), 
   by = .(main_idx), .SDcols = c('col3','col4')][
         ,.(col3 = -sum(col3), col4 = -sum(col4))]
#Result
#     col3      col4 
#0.7575290 0.2423651

Data:

DT <- data.table(main_idx = c(rep('A',4), rep('B', 5)), 
              col1 = runif(9), col2 = -2+runif(9), 
              col3 = 1+runif(9), col4 = 1+runif(9), y = runif(9))

answered Apr 09 '18 at 18:09

MKR

19,739
4
23
33

I forgot to mention that I may have dynamic number of columns to aggregate (e.g. right now I have just col3, col4. I can also have col5, col6...) – deepAgrawal Apr 09 '18 at 18:16
1

Nevermind. I can use .SD to do the dynamic number of columns – deepAgrawal Apr 09 '18 at 18:22
That's fine. All you need to vary is `.SDcols`. May be in 2nd `[` operator just exclude `main_idx` and get the `sum`. You can set the column name anyway. – MKR Apr 09 '18 at 18:22
@deepAgrawal exactly. – MKR Apr 09 '18 at 18:22
But still the major computation is being done in the lapply group by part. Is there a way to make that faster? – deepAgrawal Apr 09 '18 at 18:23
Yes. It will switch to `lapply` context but only once per column. – MKR Apr 09 '18 at 18:28

Faster aggregate multiple columns

1 Answers1