0

I have a sparse matrix having more than 200k rows and 200k columns. When I use

as.data.frame(as.matrix(sparse_matrix)

I get the following error message

Error: cannot allocate vector of size 1232.3 Gb In addition: Warning message: In asMethod(object) : sparse->dense coercion: allocating vector of size 1232.3 GiB

jpsmith
  • 11,023
  • 5
  • 15
  • 36
Vortenzie
  • 75
  • 9
  • 1
    This may be helpful: [R memory management / cannot allocate vector of size n Mb](https://stackoverflow.com/questions/5171593/r-memory-management-cannot-allocate-vector-of-size-n-mb) – jpsmith Jan 12 '23 at 18:39
  • 1
    If you have 200K rows and 200K columns then you would need a data frame with 40 billion entries. Assuming each of these only requires 8 bytes to store, then you would need at least 320 Gigabytes of memory to have a data frame that large. I'm not sure why R needs even more than that here, but in any case you are asking R to create an object that is far too large for your computer to hold in memory. Why do you need a data frame that large? – Allan Cameron Jan 12 '23 at 19:09
  • Well, I want to be able to perform regression on the data. If I try performing logistic regression on a sparse matrix, I get the following error -> Error in as.data.frame.default(data) : cannot coerce class ‘structure("ngCMatrix", package = "Matrix")’ to a data.frame – Vortenzie Jan 12 '23 at 20:04
  • 2
    `glmnet` will do logistic regression on sparse data matrices IIRC, so just use that – CJR Jan 13 '23 at 15:04
  • Thank you, it worked but I also need to use neural network on that data size. What do I do? – Vortenzie Jun 08 '23 at 18:32

0 Answers0