0

I am using the mediation package to run a mediation analysis.

But when I call the mediate function, R eats up all RAM, then it eats up all swap, and then RStudio suddenly closes.

The same happens if I call Rscript from the terminal, and the terminal suddenly closes.

Here is measures for RAM and swap. Notice the peaks:

enter image description here

This happens both with the R 4.0.5 version that comes with the Fedora repository (which uses Open BLAS), both with a custom R 4.1.0 version I compiled from source against Intel MKL.

What is the cause of this, and how I can debug?

Here is my current sessionInfo():

> sessionInfo()
R version 4.1.0 (2021-05-18)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Fedora 34 (Workstation Edition)

Matrix products: default
BLAS/LAPACK: /opt/intel/oneapi/mkl/2021.2.0/lib/intel64/libmkl_gf_lp64.so.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_GB.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_GB.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_GB.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.1.0 tools_4.1.0
robertspierre
  • 3,218
  • 2
  • 31
  • 46
  • 1
    Why do you assume there is a bug? I would assume your task is simply too big for your system's resources. – Roland Jun 22 '21 at 10:10
  • @Roland yeah, I think this is the case too, unfortunately .... – robertspierre Jun 22 '21 at 22:04
  • But is it normal that it just *shuts down*? Shoudn't it throw a message saying "I need XX GB of RAM. You have YY < XX available. So goodbye" or something like that? Why it just shuts down suddenly? – robertspierre Jun 22 '21 at 22:05
  • Your are on a linux system. Memory is managed by the OS and you need to set a memory limit: https://rdrr.io/r/base/Memory-limits.html and https://stackoverflow.com/a/24475329/1412059 – Roland Jun 23 '21 at 05:21
  • A bit of self promotion here, but if you are looking for some alternative, the `JSmediation` package also offers a way to fit mediation models in `r`. Performance-wise, it seems that it performs better in some scenario (e.g., computing conditional indirect effect in moderated mediation; https://github.com/cedricbatailler/JSmediation/pull/17#issuecomment-872287586). The number of supported models might be limited though. – Cedric Jul 06 '21 at 14:44
  • @Cedric can I fit a `lm` for `mediator~covariates` and a `glm` with a `logit` link function for `dependent~mediator+covariates` with `JSmediation`? – robertspierre Jul 06 '21 at 17:55

1 Answers1

0

If the application needs all RAM and all virtual RAM you definitely hit the hard limit of your computer. If the application now requests more RAM from the OS, that request has to fail. How do you expect the application to deal with it?

As we do not know why more RAM was requested (it could be a memory leak, it could be that the algorithm just requires that much) we cannot distinguish bug or feature, but we can try things around that.

One quite cheap thing to try is to increase virtual memory size. Execution will get slower (disk access is measured in ms as opposed to RAM access times measured in ns), but maybe you find a value that the application can complete the task with.

If you cannot find any limit on virtual memory and the application always crashes due to the out-of-memory situation, you are likely facing a bug.

In any case you may want to talk to the vendor about this perception.

Queeg
  • 7,748
  • 1
  • 16
  • 42