Efficient Empirical Distribution Calculation

Question

Consider empirically estimating the conditional distribution discrete in both X and Y,

Pr(Y|X)

Both variables have been mapped to integer sets such that

X in {1, ..., N_X} and Y in {1, ..., N_Y}

I have a dataframe of observations obs, such that obs$x[t] and obs$y[t] are my observed X and Y values for event t.

My question then is, what is the most efficient way to convert obs into a matrix F containing the empirical distributions such that

F[i,j] = sum((obs$x == i) & (obs$y == j))/sum(obs$x == i)

Of course I can use a double for loop for i in (1:N_X) and j in (1:N_Y) but I'm looking for the most efficient way.

chinsoon12 · Answer 1 · 2017-04-05T10:01:48.113

here is a method using data.table which probably can be optimized further

#data
library(data.table)
Nx <- 1e3
Ny <- 1e2
num <- 1e4
set.seed(1L)
obs <- data.table(t=1:num, 
    x=sample(1:Nx, num, replace=TRUE),
    y=sample(1:Ny, num, replace=TRUE))

#calculate F_{i,j}
ans <- obs[, {
        n = .N
        .SD[, list(Fxy=.N/n), by=.(y)]
    }, by=.(x)]

#convert into matrix
library(Matrix)
matAns <- as.matrix(sparseMatrix(
    i=ans[["x"]], 
    j=ans[["y"]],
    x=ans[["Fxy"]]
))

head(matAns)

would love to learn a faster method to calculate this

Efficient Empirical Distribution Calculation

1 Answers1