0

rbind fails on these two sparse matrices each, 320k rows x 31k columns, with the error that "problem too large".

I was hoping someone might have some suggestions. One example below first shows attributes of each matrix, system memory, and then tries to bind them using rbind and rbind2(,sparse=T):

enter image description here

Alon Gelber
  • 113
  • 7

1 Answers1

1

You could maybe try dplyr and the bind_rows() function? https://dplyr.tidyverse.org/reference/bind.html

library(dplyr)
mat3 <- bind_rows(mat1, mat2)

I don't understand the error message, and it would be surprising if it was an issue due to the size of the dataset. Just to be sure, you could try binding and inspecting whether it works for a very small subset of your matrices:

rbind(mat1[1:3, ], mat2[1:3, ])
Rosalie Bruel
  • 1,423
  • 1
  • 10
  • 22
  • Apparently, the matrices are not all that sparse (about 10% non-zero) and sparse matrices in R can only hold about 2^31 non-zero elements. I'd like to combine 4 of about the same size – Alon Gelber Mar 13 '21 at 07:40