0

I was curious on how I can turn this to an Rcpp format? Because of memory problem, R's as.matrix method isn't working 

sparse.cor4 <- function(x){
    n <- nrow(x)
    cMeans <- colMeans(x)
    covmat <- (as.matrix(crossprod(x)) - n*tcrossprod(cMeans))/(n-1)
    sdvec <- sqrt(diag(covmat)) 
    cormat <- covmat/tcrossprod(sdvec)
    list(cov=covmat,cor=cormat)
}

The function is from this link: Running cor() (or any variant) over a sparse matrix in R

Additional information: I was able to create a sparse matrix of 500kx500k using rcpp but I need to correlate that sparse matrix which wasn't possible with R's cor() because of memory size which is why I am asking if I can convert the above function into Rcpp mode to get the correlation of the sparse matrix

Thanks

Viktor
  • 47
  • 5
  • 1
    Rcpp does not change memory requirements. A double is stil eight bytes and 64 bits. Exactly what type of magic are you hoping for? – Dirk Eddelbuettel Oct 27 '19 at 05:11
  • in Rcpp, I am able to create large matrices that R's Matrix package can't handle using ```ARMA_64BIT_WORD```, so I thought there was something like that possible for the above function in rcpp – Viktor Oct 27 '19 at 05:27
  • 2
    Can you give an example for the size and type of matrix you want to use this for? Can you give an example matrix you can create in C++ but not in R? You can [edit] your question to include the additional information. – Ralf Stubner Oct 27 '19 at 09:58
  • I have added extra information. thanks – Viktor Oct 27 '19 at 12:58
  • 1
    This will be very difficult. The correlation matrix for a nxp matrix is a dense pxp matrix. The linked question was for a sparse nxp matrix with n >> p. So making sure one only had dense pxp matrices did the trick. You have a square matrix, so this will not help you. And for a dense 500k x 500k matrix you will need something like 2 TB of memory. – Ralf Stubner Oct 27 '19 at 15:26
  • Yes, I saw that it was changed to a dense matrix, could that be achievable using rcpp? changing the sparse matrix to a dense matrix but in rcpp. what other suggestions can you make in me getting it to work? – Viktor Oct 27 '19 at 15:37
  • 1
    [Armadillo](http://arma.sourceforge.net/docs.html#Mat) has a constructor for converting sparse to dense. However, this does not solve the fundamental memory problem. A 500k x 500k matrix is still 2 TB large if you have it in C++ as opposed to R. BTW, it makes sense to @-mention those you are replying. to. – Ralf Stubner Oct 28 '19 at 08:59

0 Answers0