"Too many indices" big matrix vector length issue in R

Question

Hello and thanks in advance. I'm using Rx64 version 3.1.2 on a Windows Server and have a file-backed big matrix generated from the package bigmemory that I'm trying to use in a linear programming problem. The matrix is 7062 rows by 364520 columns for a total of 2574240240 entries (of integers).

When I run the line for the linear program, I get the following error:

Error in GetElements.bm(x, i, j) : 
  Too many indices (>2^31-1) for extraction.

That number, 2147483647, from what I read is the maximum number of entries R allows for any object even on 64-bit processes. I've read here that I can use the 'experimental version' of R to get around this but I was hoping there's a recently available solution. I have supporting output below to confirm my R version:

> version
               _                           
platform       x86_64-w64-mingw32          
arch           x86_64                      
os             mingw32                     
system         x86_64, mingw32             
status                                     
major          3                           
minor          1.2                         
year           2014                        
month          10                          
day            31                          
svn rev        66913                       
language       R                           
version.string R version 3.1.2 (2014-10-31)
nickname       Pumpkin Helmet 

> Sys.getenv("R_ARCH")
[1] "/x64"

Long vectors are supported in R since version 3, so perhaps the issue lies with bigmemory. — James, Mar 03 '15 at 16:12
The error is hard-coded into bigmemory.cpp: http://goo.gl/QYQfga — James, Mar 03 '15 at 16:21
@James so if I edit bigmemory.cpp and remove all mention of that hard-coded error will it solve the problem? — gtnbz2nyt, Mar 03 '15 at 16:28
It *might*. I would suggest emailing the package author for his views. — James, Mar 03 '15 at 16:49

score 1 · Accepted Answer · answered Mar 03 '15 at 19:22

It would be best if you could also provide the 'line for the linear program' but my comment is too long to put above.

@James is correct that 32-bit is hard-coded in to bigmemory at the moment. After glancing at the source on github I see the author did remove the constraint and you can download that newest version with:

library(devtools)
install_github("kaneplusplus/bigmemory")

Without that limitation I believe you will still have a problem. If I assume correctly, you are also using the complementary package bigalgebra. There is a way to use 64-bit with bigalgebra by reintstalling the downloaded tar ball and installing with:

REFBLAS=1 R CMD INSTALL bigalgebra_0.8.4.tar.gz

NOTE - the functions available for 64-bit are very limited currently and will be slower than the non-64-bit versions because of the currently workarounds.

I have been in contact with the original author and I am in the process of updating the packages to utilize RcppArmadillo which will provide a cleaner interface and deal with the 64-bit problem. This is currently in progress at my fork of the package on my github account.

following the advice of 'James' I contacted the author of bigmemory and he did in fact make the change. Also I'm not using bigalgebra, I'm actually using the Rsymphony package and the Rsymphony_solve_LP function. I can edit my question with more details but this might be worth an entire new question... — gtnbz2nyt, Mar 03 '15 at 19:57

score 0 · Answer 2 · answered Mar 03 '15 at 16:16

0

The best you can do is basically divide and conquer, meaning splitting your matrix and do as many steps (more) as necessary to reach your goal.

answered Mar 03 '15 at 16:16

Adonis

4,670
3
37
57

"Too many indices" big matrix vector length issue in R

2 Answers2

Linked