0

I have a correlation matrix in R for >25000 variables. The problem is, that I can't do anything with it because of the huge amount of needed memory. What I don't understand why I can't even to very primitive stuff like modifing single entries. E.g.

> cors = cor(my.data)
> class(cors)
[1] "matrix"
> dim(cors)
[1] 26267 26267
> cors[1,1] = 1
Error: cannot allocate vector of size 5.1 Gb

Why should R need that much memory to change one value...? I get the same errors when I try stuff like diag(cors) or upper.tri(cors)...I know that sparse matrices might offer a solution but when I try to convert it i get the same error....

> m = Matrix(cors, sparse=T)
Error: cannot allocate vector of size 5.1 Gb

Why is it so difficult even when I want to modify a single value only?

Jonas
  • 1,639
  • 1
  • 18
  • 29
  • 4
    The whole matrix needs to be copied if you change a single value of it. Package `data.table` offers assignment by reference, but I doubt somewhat that you can transform a matrix into a data.table without a copy. You could work in chunks or try using a package like bigmemory. I don't think sparse matrices will help, because exactly 0 correlation is not very likely. – Roland Jan 16 '15 at 10:00
  • Maybe [this](http://stackoverflow.com/questions/5171593/r-memory-management-cannot-allocate-vector-of-size-n-mb) will help. Apart from `bigmemory` that Roland suggests, you could also try package `ff` (which in my experience worked well) – LyzandeR Jan 16 '15 at 10:03
  • 1
    Possible duplicate of [R memory management / cannot allocate vector of size n Mb](https://stackoverflow.com/questions/5171593/r-memory-management-cannot-allocate-vector-of-size-n-mb) – Benjamin May 23 '18 at 01:51

0 Answers0