Generate random sample using weights by group

Question

I am trying to randomly sample a data.table using groups. The sample size of each group will be computed by multiplying the frequency with Sample_Size, which is the expected number of rows in the output data.table.

I researched this topic on SO. It seems similar threads (Need to randomly sample a data set with multiple groups each with multiple factors and take randomly sample based on groups) have assumed uniform distribution for weights, which doesn't work for me.

Here's test data:

InputDT <- data.table::data.table ("Country"=c(rep("A",20),rep("B",10),rep("C",5),rep("D",2)), "ID"=c(1:20,101:110,201:205,301:302))

The objective is to sample IDs by country.

Here's the frequency we want:

CountryFreq <- 
 data.table::data.table("Country"=unique(InputDT$Country), "Freq"=c(4/10,2/10,2/10,2/10))

Here's the number of rows in the output data.table:

 Sample_Size <- 10

As a rule, let's assume that Sample_Size < nrows(InputDT)

Here's manually created sample output:

OutputDT <- structure(list(Country = c("A", "A", "A", "A", "B", "B", "C", 
"C", "D", "D"), ID = c(1, 5, 7, 3, 102, 109, 203, 204, 301, 302
)), .Names = c("Country", "ID"), row.names = c(NA, 10L), class = "data.frame")

Here's a test to check whether frequencies are as needed:

Hmisc::describe(OutputDT$Country)

OutputDT$Country 
       n  missing distinct 
      10        0        4 

Value        A   B   C   D
Frequency    4   2   2   2
Proportion 0.4 0.2 0.2 0.2

Can someone please help me? I've spent almost one day trying to learn sampling in R and then customizing it to my need. I'd appreciate any help.

Answer 1

We can do

InputDT[, rbindlist(Map(function(x, y) x[sample(seq_len(nrow(x)), y)], 
        split(.SD, Country), freq))]

data

freq <- c(4, 2, 2, 2)

Answer 2

You can join InputDT with country frequency first before doing the sampling for each country as follows:

InputDT[CountryFreq,
    .SD[sample(.N, min(.N, Freq*Sample_Size))], 
    by=.EACHI,
    on=.(Country)]

Notes:

InputDT[i=CountryFreq, on=.(Country)] joins CountryFreq with InputDT using Country as the key.

by=.EACHI performs this j=.SD[sample(.N, min(.N, Freq*Sample_Size))] for each row in i=CountryFreq. Note by=.EACHI only works for an equi-join for now.

.SD is Subset of Data of InputDT, i.e. each subset of data from InputDT for each Country in each row of i because of by=.EACHI. .SD is only really within the scope of InputDT and can only be used in j. See ?data.table. To learn more, check out lexical scoping. A good reference is Advanced R by Hadley Wickham.

sample(.N, min(.N, Freq*Sample_Size)) samples Freq*Sample_Size indices from the number of rows from .SD while min ensures that you dont sample more than available samples in that country.

Finally, .SD[sample(.N, min(.N, Freq*Sample_Size))] subset sampled rows of .SD.

---

edit: show sample runs from R console.

> InputDT[CountryFreq,
+     .SD[sample(.N, min(.N, Freq*Sample_Size))], 
+     by=.EACHI,
+     on=.(Country)]
    Country  ID
 1:       A  19
 2:       A   7
 3:       A   5
 4:       A   3
 5:       B 109
 6:       B 110
 7:       C 203
 8:       C 205
 9:       D 302
10:       D 301
> InputDT[CountryFreq,
+     .SD[sample(.N, min(.N, Freq*Sample_Size))], 
+     by=.EACHI,
+     on=.(Country)]
    Country  ID
 1:       A  12
 2:       A  19
 3:       A  17
 4:       A  10
 5:       B 110
 6:       B 105
 7:       C 202
 8:       C 203
 9:       D 302
10:       D 301
> InputDT[CountryFreq,
+     .SD[sample(.N, min(.N, Freq*Sample_Size))], 
+     by=.EACHI,
+     on=.(Country)]
    Country  ID
 1:       A   9
 2:       A   7
 3:       A  19
 4:       A   6
 5:       B 106
 6:       B 108
 7:       C 205
 8:       C 201
 9:       D 302
10:       D 301