Cartesian Rolling Join using Data.table

Question

I have two tables:

dat: contains the data
dates: contains the table of dates

library(data.table)

dates = structure(list(date = structure(c(17562, 17590, 17621, 17651, 
                              17682, 17712, 17743, 17774, 17804, 17835, 17865, 17896), class = "Date")), 
      row.names = c(NA, -12L), class = "data.frame")


dat = structure(list(date = structure(c(17546, 17743, 17778, 17901, 
                              17536, 17806, 17901, 17981, 17532, 17722, 17969, 18234), class = "Date"), 
           country = structure(c(1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 3L, 
                                 3L, 3L, 3L), .Label = c("AAA", "BBB", "CCC"), class = "factor"), 
           state = structure(c(1L, 1L, 2L, 3L, 4L, 1L, 2L, 5L, 6L, 1L, 
                               2L, 2L), .Label = c("S1", "S2", "S3", "S4", "S5", "S6"), class = "factor"), 
           item = structure(c(1L, 2L, 4L, 6L, 3L, 5L, 3L, 2L, 2L, 4L, 
                              5L, 7L), .Label = c("M1", "M2", "M3", "M4", "M5", "M6", "M7"
                              ), class = "factor"), value = c(67L, 10L, 50L, 52L, 93L, 
                                                              50L, 62L, 46L, 6L, 30L, 30L, 14L)), row.names = c(NA, -12L
                                                              ), class = "data.frame")


dates = data.table(dates)
dat = data.table(dat)


setkey(dates, date)
setkey(dat, date)

The result I'm after is below. I.e doing a rolling join with each individual row of dat and then combining the result.

rbind(
dat[1,][dates, roll = 90],
dat[2,][dates, roll = 90],
dat[3,][dates, roll = 90],
...
dat[12,][dates, roll = 90]
)

My actual dataset is much larger so it's no practical to list every row of dat. Is there a short hand way of doing the same thing without a loop?

score 1 · Accepted Answer · answered Jun 19 '18 at 07:11

If I understand your intent correctly, you want to rollover the records for 90 days. I used a cross join and then used the rollover criteria to subset

Your original tables:

library(data.table)

dates = structure(list(date = structure(c(17562, 17590, 17621, 17651, 
                                          17682, 17712, 17743, 17774, 17804, 17835, 17865, 17896), class = "Date")), 
                  row.names = c(NA, -12L), class = "data.frame")


dat = structure(list(date = structure(c(17546, 17743, 17778, 17901, 
                                        17536, 17806, 17901, 17981, 17532, 17722, 17969, 18234), class = "Date"), 
                     country = structure(c(1L, 1L, 1L, 1L, 2L, 2L, 2L, 2L, 3L, 
                                           3L, 3L, 3L), .Label = c("AAA", "BBB", "CCC"), class = "factor"), 
                     state = structure(c(1L, 1L, 2L, 3L, 4L, 1L, 2L, 5L, 6L, 1L, 
                                         2L, 2L), .Label = c("S1", "S2", "S3", "S4", "S5", "S6"), class = "factor"), 
                     item = structure(c(1L, 2L, 4L, 6L, 3L, 5L, 3L, 2L, 2L, 4L, 
                                        5L, 7L), .Label = c("M1", "M2", "M3", "M4", "M5", "M6", "M7"
                                        ), class = "factor"), value = c(67L, 10L, 50L, 52L, 93L, 
                                                                        50L, 62L, 46L, 6L, 30L, 30L, 14L)), row.names = c(NA, -12L
                                                                        ), class = "data.frame")


dates = data.table(dates)
dat = data.table(dat)

Note, I haven't setkey.

I am using a cross join function from the reference: How to do cross join in R?

CJ.table.1 <- function(X,Y)
  setkey(X[,c(k=1,.SD)],k)[Y[,c(k=1,.SD)],allow.cartesian=TRUE][,k:=NULL]

Then I cross join, subset for the roll join, rename columns and sort

dsn1<-CJ.table.1(dat,dates)[i.date-date<=90 & i.date-date>=0][,.(date=i.date,country, state, item, value)][order(country, state, item, value,date),]

Same idea, I think: `dat[dates[, .(k = seq(1L, nrow(dat))), by=date], roll=90, allow.cartesian=TRUE][order(k)][, k := NULL][]` — Frank, Jun 19 '18 at 23:36

score 0 · Answer 2 · answered Jun 19 '18 at 03:48

This is not necessarily the best way to do it, but you could simply write a loop here to iterate through your data:

df <- data.frame()

for (i in 1:nrow(dat)){
    df <- rbind(df, dat[i,][dates, roll = 90])
}

head(df)

          date country state item value
  1: 2018-01-31     CCC    S6   M2     6
  2: 2018-02-28     CCC    S6   M2     6
  3: 2018-03-31     CCC    S6   M2     6
  4: 2018-04-30    <NA>  <NA> <NA>    NA
  5: 2018-05-31    <NA>  <NA> <NA>    NA

Edit: just saw you said "without a loop", it's been a long day. This is one way to solve the problem though.

Dynamically growing objects in R via rbind or similar is generally very inefficient (I guess you know). I don't think a loop's necessarily terrible, though I'd just initialize df to have the right number of rows and correct set of cols, like eg `iris[rep(NA_integer_, 10), ]`, then assign to each row instead of growing it. — Frank, Jun 19 '18 at 23:28

Cartesian Rolling Join using Data.table

2 Answers2