0

In R, I'm trying to row-bind several identity matrices into 1 giant matrix using the code below:

>     X <- 4
>     Y <- 3
>     block1 <- diag(X)   
>     for(x in 2:Y) {
>     block1 <- cbind(block1,diag(X))   
>        }

Should look like this:

     [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12]
[1,]    1    0    0    0    1    0    0    0    1     0     0     0
[2,]    0    1    0    0    0    1    0    0    0     1     0     0
[3,]    0    0    1    0    0    0    1    0    0     0     1     0
[4,]    0    0    0    1    0    0    0    1    0     0     0     1

I feel like there is a cleaner and less memory taxing way to do this. Truth is my X is in the several thousands and Y is in the 20s so generating the matrix 'from scratch' without using these replacements in the for-loop would be preferred because I'm getting memory blockages. I tried looking at bdiag() in the Matrix package but that function isn't exactly what I want.

So is there a function that would do this 'from scratch'?

EDIT:

Users were kind enough to suggest I look into 'sparse Matrices.' I came across the SparseM package and am 99% there. I simply use the as.matrix.csr function to convert my matrices into CSR format and together with both cbind.matrix.csr and rbind.matrix.csr I can efficiently get all the matrices I need. Now the issue is when I try to use it in a Linear Programming solver (Rsymphony) I have to use as.matrix() to convert the CSR matrix back to normal and it's still giving me a memory issue.

gtnbz2nyt
  • 1,465
  • 3
  • 17
  • 33

1 Answers1

2

As @OChristiaanse suggests, a sparse matrix may be the way to go. You could try this short call and see if it works for you.

library(Matrix)
do.call(cBind, replicate(Y, Diagonal(X)))
4 x 12 sparse Matrix of class "dgCMatrix"

[1,] 1 . . . 1 . . . 1 . . .
[2,] . 1 . . . 1 . . . 1 . .
[3,] . . 1 . . . 1 . . . 1 .
[4,] . . . 1 . . . 1 . . . 1

Seeing how you are also using non-sparse data (i.e. not zeros/ones) and have memory issues, you may want to look in to the bigmemory package. With these objects, you can use filebacking and have objects bigger than available memory.

EDIT

After speaking with the bigmemory maintainer I have learned you can install it for Windows. It is not on CRAN because the BH package (which it depends on) generates a warning from the C++ boost library. To install it on Windows, you need to install Rtools and then install from the github repo.

devtools::install_github("kaneplusplus/bigmemory")
cdeterman
  • 19,630
  • 7
  • 76
  • 100
  • Thanks for the guidance. I tried doing this method, then had issues when I wanted to column-bind this to another matrix (of all zeros), and there are other non-zero matrices that I need to bind this to. – gtnbz2nyt Feb 17 '15 at 18:18
  • is bigmemory available for windows? I'm looking on its CRAN page and it says the Windows binaries aren't available. 'bigmemory.sri' is available is that the same thing? – gtnbz2nyt Feb 17 '15 at 18:40
  • Ah, that is unfortunate. Yes, it is currently only for Linux right now. I am not sure at the moment what the reason behind that is. I will need to think above some other memory saving approaches unless someone else jumps in. – cdeterman Feb 17 '15 at 18:45
  • @gtnbz2nite, see my edit regarding how to get `bigmemory` on windows. Let me know how it works out :) – cdeterman Feb 18 '15 at 14:18
  • Awesome thanks for going the extra mile it's appreciate! I'll get back to you – gtnbz2nyt Feb 18 '15 at 15:08
  • alright sorry that took some time but I finally was able to take advantage of bigmemory and create my matrix no problem. Now the problem is I want to use this matrix in a function, but it looks like I'm getting an error that the number of indicies is greater than R's limit of (2^31-1). Tried looking for a solution but not getting anywhere except this thread: http://stackoverflow.com/questions/10640836/max-length-for-a-vector-in-r – gtnbz2nyt Mar 02 '15 at 23:00
  • @gtnbz2nite I am glad you managed to create the matrix. However, I would need more information to try and troubleshoot that problem. The only guess at the moment is making sure your are using a recent 64-bit R version (>3.0). I suggest opening a new question with the new function you are trying to use on this large matrix and then I could be of more help. Also, if this answer has solved your initial question, please accept it so it can be marked as closed. – cdeterman Mar 03 '15 at 13:48