0

I just got a new Acer Swift laptop and installed ubuntu 20.04 on it. I installed R.4.0.3 and Rstudio 1.2.5042

I tried to run a script that is running without problems on my old computer. R crashes and Rstudio aborts when I assign one matrix to another. I tried increasing R memory (it is inf now) and doing the same assignment using matrix subsets. It still crashes. R session is also aborted if I skip the matrix assignment and just try to do other manipulations with the matrix ext_data.df

FYI: data.input is a matrix 33538 x 11366. ext_data.df is properly initialiozed. It is only at the assignment

ext_data.df[common_genes,] = data.input[common_genes,]

that the crash happens.

ext_data.df=matrix(0,dim(gene_list)[1],dim(data.input)[2])  
rownames(ext_data.df) = gene_list$X2
colnames(ext_data.df) <- colnames(data.input)
common_genes = intersect(gene_list$X2,rownames(data.input))
ext_data.df[common_genes,] = data.input[common_genes,]


sessionInfo()

R version 4.0.3 (2020-10-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.1 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-serial/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-serial/liblapack.so.3

locale:
 [1] LC_CTYPE=en_SG.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_SG.UTF-8        LC_COLLATE=en_SG.UTF-8    
 [5] LC_MONETARY=en_SG.UTF-8    LC_MESSAGES=en_SG.UTF-8   
 [7] LC_PAPER=en_SG.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_SG.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] tensorflow_2.2.0  kknn_1.3.1        plotly_4.9.3      viridis_0.5.1    
 [5] viridisLite_0.3.0 KernelKnn_1.1.0   forcats_0.5.0     stringr_1.4.0    
 [9] dplyr_1.0.3       purrr_0.3.4       readr_1.4.0       tidyr_1.1.2      
[13] tibble_3.0.5      tidyverse_1.3.0   keras_2.3.0.0     ggpubr_0.4.0     
[17] ggplot2_3.3.3     ggsci_2.9         class_7.3-17      Seurat_3.2.3     

loaded via a namespace (and not attached):
  [1] readxl_1.3.1         backports_1.2.1      plyr_1.8.6          
  [4] igraph_1.2.6         lazyeval_0.2.2       splines_4.0.3       
  [7] listenv_0.8.0        scattermore_0.7      tfruns_1.4          
 [10] digest_0.6.27        htmltools_0.5.1.1    fansi_0.4.2         
 [13] magrittr_2.0.1       tensor_1.5           cluster_2.1.0       
 [16] ROCR_1.0-11          openxlsx_4.2.3       globals_0.14.0      
 [19] modelr_0.1.8         matrixStats_0.57.0   askpass_1.1         
 [22] colorspace_2.0-0     rvest_0.3.6          rappdirs_0.3.1      
 [25] ggrepel_0.9.1        haven_2.3.1          crayon_1.3.4        
 [28] jsonlite_1.7.2       spatstat_1.64-1      spatstat.data_1.7-0 
 [31] zeallot_0.1.0        survival_3.2-7       zoo_1.8-8           
 [34] glue_1.4.2           polyclip_1.10-0      gtable_0.3.0        
 [37] leiden_0.3.7         car_3.0-10           future.apply_1.7.0  
 [40] abind_1.4-5          scales_1.1.1         DBI_1.1.1           
 [43] rstatix_0.6.0        miniUI_0.1.1.1       Rcpp_1.0.6          
 [46] xtable_1.8-4         reticulate_1.18      foreign_0.8-81      
 [49] rsvd_1.0.3           umap_0.2.7.0         htmlwidgets_1.5.3   
 [52] httr_1.4.2           RColorBrewer_1.1-2   ellipsis_0.3.1      
 [55] ica_1.0-2            pkgconfig_2.0.3      uwot_0.1.10         
 [58] dbplyr_2.0.0         deldir_0.2-9         tidyselect_1.1.0    
 [61] rlang_0.4.10         reshape2_1.4.4       later_1.1.0.1       
 [64] munsell_0.5.0        cellranger_1.1.0     tools_4.0.3         
 [67] cli_2.2.0            generics_0.1.0       broom_0.7.3         
 [70] ggridges_0.5.3       fastmap_1.1.0        goftest_1.2-2       
 [73] fs_1.5.0             fitdistrplus_1.1-3   zip_2.1.1           
 [76] RANN_2.6.1           pbapply_1.4-3        future_1.21.0       
 [79] nlme_3.1-151         whisker_0.4          mime_0.9            
 [82] xml2_1.3.2           compiler_4.0.3       rstudioapi_0.13     
 [85] curl_4.3             png_0.1-7            ggsignif_0.6.0      
 [88] spatstat.utils_2.0-0 reprex_0.3.0         stringi_1.5.3       
 [91] RSpectra_0.16-0      lattice_0.20-41      Matrix_1.3-2        
 [94] vctrs_0.3.6          pillar_1.4.7         lifecycle_0.2.0     
 [97] lmtest_0.9-38        RcppAnnoy_0.0.18     data.table_1.13.6   
[100] cowplot_1.1.1        irlba_2.3.3          httpuv_1.5.5        
[103] patchwork_1.1.1      R6_2.5.0             promises_1.1.1      
[106] KernSmooth_2.23-18   gridExtra_2.3        rio_0.5.16          
[109] parallelly_1.23.0    codetools_0.2-18     MASS_7.3-53         
[112] assertthat_0.2.1     openssl_1.4.3        withr_2.4.0         
[115] sctransform_0.3.2    mgcv_1.8-33          parallel_4.0.3      
[118] hms_1.0.0            grid_4.0.3           rpart_4.1-15        
[121] carData_3.0-4        Rtsne_0.15           shiny_1.6.0         
[124] lubridate_1.7.9.2    base64enc_0.1-3

Could you please help me solve the problem?

Thank you very much!

Phil
  • 7,287
  • 3
  • 36
  • 66
Eleni Christodoulou
  • 21
  • 4
  • "I tried increasing R memory (it is inf now)" A computer cannot have infinite memory. I would suggest limiting memory (probably by setting `R_MAX_VSIZE`) to ensure a graceful error that doesn't kill the process. I would also use system tools and monitor memory usage up to the process being killed. – Roland Jan 27 '21 at 07:13
  • I tried Sys.setenv('R_MAX_VSIZE'=32000000000). The same thing happened. I don't think it is a memory problem, because I tried the same command after subsetting the matrices to include only the first 10 rows: `ext_data.df[common_genes[1:10,] = data.input[common_genes[1:10,] ` – Eleni Christodoulou Jan 27 '21 at 08:31
  • How long is `common_genes`? Can you try without 125 packages being loaded? – Roland Jan 27 '21 at 13:52
  • common_genes is 22400 long. I am working with high dimensional gene expression data. I need all these packages in my script. As I said, this works perfectly on other computers. Could it be something related to the BLAS or LAPACK matrix products? – Eleni Christodoulou Jan 28 '21 at 01:55

3 Answers3

0

Could it be that you are using 32 bit version of R? 32 bit applications can only access a maximum of 4 Gigs of memory. Can you try the following

.Machine$sizeof.pointer

it should return 8 if you are using 64 bit R

Carlos Santillan
  • 1,077
  • 7
  • 8
  • Thank you very much Carlos! It returns 8 indeed. I noticed that I have other problems with the matrices data.input and ext_data.df. For example, I cannot apply functions like summary() on them. Any clues? – Eleni Christodoulou Jan 28 '21 at 03:46
0

Update: It is clearly a memory problem. When I try to do the assignment ext_data.df[common_genes,1:1000]=data.input[common_genes, 1:1000] it works. I monitored the usage with mem_used() and found that it works up to 7.8GB (it happens for ext_data.df[common_genes,1:3000]=data.input[common_genes, 1:3000]. At 8GB, the R session gets aborted. How can I increase the memory above 8GB. In my old laptop (MacBook Pro), the mem_used was 8.21GB for the whole assignment. I have to note that I use Sys.setenv('R_MAX_MEM_SIZE'=64000000000) and Sys.setenv('R_MAX_VSIZE'=64000000000) in the beginning of my script.

Eleni Christodoulou
  • 21
  • 4
  • How much physical RAM does this machine have? Is memory limited by the OS (e.g., using `limit` or [`ulimit`](https://linuxhint.com/linux_ulimit_command/))? – Roland Jan 28 '21 at 06:54
  • `ulimit` returns `unlimited`...I really have no clue. My machine came with 16GB of memory – Eleni Christodoulou Jan 28 '21 at 07:52
0

Maybe you can try increasing the memory via memory.limit().

You can refer to this stackoverflow thread for more information: Increasing (or decreasing) the memory available to R processes

imouyang
  • 1