1

I need a memory shared sparse matrix of 1.3e6x200. I have tried with bigmemory package but my computer ran out of memory. I have found the function FBM in the package bigstatsr. My code run pretty good with this function for a small example, but I cannot create a matrix higher than 150 000 rows. Any idea could I solve this? Thank you for your time This is the error I have trying to use FBM function

LauC
  • 55
  • 5
  • What do you mean by "I cannot create a matrix higher than 150 000 rows"? I recently created a matrix 450,000 x 560,000. – F. Privé Oct 02 '18 at 18:45
  • > FBM(nrow=1.3e4,ncol=1.3e6) Error in getXPtrFBM(.self$bk, .self$nrow, .self$ncol, .self$type) : El volumen para un archivo ha sido alterado externamente, por lo que el archivo abierto ya no es válido. In English: the volume of the file has been changed externally and the open file is not valid – LauC Oct 03 '18 at 09:17
  • Have you enough free space on your disk? Have you tried specifying a non-temporary `backingfile`? – F. Privé Oct 03 '18 at 10:11
  • Error in getXPtrFBM(.self$bk, .self$nrow, .self$ncol, .self$type) : The volume for a file has been externally altered so that the opened file is no longer valid. > gc() used (Mb) gc trigger (Mb) max used (Mb) Ncells 1820217 97.3 3478931 185.8 3052899 163.1 Vcells 210097560 1603.0 361882860 2761.0 301487170 2300.2 – LauC Oct 03 '18 at 13:08
  • sorry but I don´t understadnd what you mean by trying specifying a non-temporary backingfile. – LauC Oct 03 '18 at 13:10
  • There is an argument `backingfile` in `FBM()` to specifiy the location of the backingfile(s). By default, it writes in the `/tmp/` directory. Try é.g. `backingfile = "test"` and it should write a file `test.bk` that contains the data of the FBM. – F. Privé Oct 03 '18 at 15:32
  • `gc()` is showing the amount of RAM used. I'm taking about the amount of DISK space. – F. Privé Oct 03 '18 at 15:33
  • you were right I am almost run out of memory. When I use your function it gives me the same message. How could I save it in an external hard disc, for example in "G:\results\" FBM(1.3e6,1.3e6,backingfile="G:\results\fbm") – LauC Oct 04 '18 at 14:10
  • I never tried with an external disk. You're on Windows? Try e.g. to double the backslashes. Otherwise, you should clean your disk from unused files. – F. Privé Oct 04 '18 at 15:32
  • I am using windows. If I create an R project in my hard disk to run the script and I use FBM(1.3e6,128,backingfile = "test") I can create the file directly in the hard disk. Thank you very much for your help and congratulations for the package – LauC Oct 04 '18 at 17:18

0 Answers0