Problem with big_cprodMat output: Matrix too large

Question

I am trying to get the cross-product of two "Large matrices" in R. Unfortunately, the data cannot be disclosed here, but the dimensions are 177304*23, and 23*177304, so that the results should have the dimensions 177304*177304.

As my RAM is 8 GB, I have been using the FBM-class to store the matrices in the Disk Storage Space instead. But when I use the function big_cprodMat to find the crossproduct, it seems to first solve the product using a matrix class-object as the output and R crashes instead.

Is there anything I could do to avoid this?

I'm new to the website and greatly appreciate any help anyone could offer.

Care to provide a working example with simulated data? – Roman Luštrik Jul 24 '19 at 15:09 — Roman Luštrik, Jul 24 '19 at 15:09

score 0 · Answer 1 · answered Jul 24 '19 at 16:14

0

Using a smaller example:

N <- 17e3
M <- 23
A <- matrix(rnorm(N * M), N, M)
B <- matrix(rnorm(N * M), M, N)

You can fill the resulting matrix by blocks (note that it will take 234 GB on disk for the size you ask for):

library(bigstatsr)
AB <- FBM(N, N, backingfile = "test")$save()
big_apply(AB, a.FUN = function(X, ind) {
  X[, ind] <- A %*% B[, ind]
  NULL
}, block.size = 1e3)

Do not hesitate to open an issue on the GitHub repo of {bigstatsr}.

answered Jul 24 '19 at 16:14

F. Privé

11,423
2
27
78

Thanks! The code worked, but I've opened an issue regardless to add the possibility of future improvements. – Shuvani Choudhury Jul 25 '19 at 11:44
Ok. If this works for you, please validate the answer. – F. Privé Jul 25 '19 at 11:54

Problem with big_cprodMat output: Matrix too large

1 Answers1