2

Until recently I have been using rpy2 to load an R package into a Jupyter notebook running a Python kernel, and use it to read some data from a sqlserver database. A few days ago, I started seeing this error:

import rpy2.robjects as ro
R[write to console]: Error: cons memory exhausted (limit reached?)

Following this error the Python kernel crashes and is restarted by Jupyter. I would like to understand this error, what is causing it and why it has appeared when the same code ran without errors previously.

This answer, on r-devel, suggests that it is an R error produced by exceeding a maximum number of objects:

[Rd] How to debug: Cons memory exhausted

Other than this, there seems to be very little information available about this error.

The error occurs on the first line in my code. This, and the fact that the code ran previously, makes me think something is being cached from a previous sessions. I can import the data in R (using RStudio), so I think this is specific to Jupyter/rpy2 rather than a general R issue. It is not specific to this particular notebook though - if I run the above import in another notebook, on a different kernel, I get the same error. So my questions are:

  1. Is an excess of R Objects the only thing that causes this error, or might it be something else?
  2. Is this likely a caching problem or am I on the wrong track here?
  3. Where might these objects be cached? (and how can I safely clear them?)

I am using jupyterlab in a conda venv on Windows 10 in a remote Amazon workspace. I have no admin privileges. Here are some versions:

import session_info
session_info.show()
-----
rpy2                3.4.4
session_info        1.0.0
-----
Click to view modules imported as dependencies
-----
IPython             7.16.1
jupyter_client      6.1.6
jupyter_core        4.6.3
jupyterlab          2.1.5
notebook            6.0.3
-----
Python 3.7.7 (default, May  6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)]
Windows-10-10.0.14393-SP0
-----

Searching SO reveals many ways in which importing robjects can fail, but I can only find one about this specific error (with no answers given):

RPY2 3.3.5 Errors on import

Tangentially, it seems as though this question might be germane:

Is it safe to manually delete all files in pkgs folder in anaconda python?

Is it recommended to clear out $HOME\AppData\Local\conda\conda\pkgs\cache regularly? (UPDATE: There were some quite large files in this cache folder but clearing them did not solve the issue).

UPDATE: I see the same error in Spyder, running outside the venv, so this isn't specific to jupyter.

UPDATE: I can import rpy2. A simple import rpy2 produces no error.

UPDATE:

import rpy2.situation as rps
for row in rps.iter_info():
    print(row)

gives:

    rpy2 version:
    3.4.4
    Python version:
    3.7.7 (default, May  6 2020, 11:45:54) [MSC v.1916 64 bit (AMD64)]
    Looking for R's HOME:
        Environment variable R_HOME: None
        InstallPath in the registry: C:\Program Files\R\R-3.6.1
        Environment variable R_USER: None
        Environment variable R_LIBS_USER: None
    R version:
        In the PATH: 
        Loading R library from rpy2: OK
    Additional directories to load R packages from:
    None
    C extension compilation:
        Warning: Unable to get R compilation flags.

Environment variables are not set in rpy2. Could this mean that R_NSIZE is unset, and could this cause the error?

majr
  • 263
  • 1
  • 6
  • There is very little information on your error. From the sounds of it, this sounds like your machine is running out of memory when starting an R session. Have you tried starting R by itself? Which version of R is connected to your python session? From the looks of it, they may be outdated (R is currently running version 4.1.0) and the error may be a version error. Also `rpy2` has a version 3.4.5, maybe the error is caused by `3.4.4`?. In short: Maybe try a new and updated python and/or R version with updated `rpy2` version, seei f this fixes the issue. – Oliver Jun 22 '21 at 12:16
  • Thanks @Oliver. Yes, I have tried starting R by itself (in R Studio) - no issues there. I don't *think* this is a version issue because the same code ran without errors until a few days ago, but in any case I don't have admin privileges on this platform so this is not an avenue I can explore. – majr Jun 22 '21 at 12:24
  • Odd, and it is immediately as you try to start an R session through `rpy2`? You are likely able to install R itself in your user folder (classic means of getting past Admin privileges). Also curious whether this also crashes a bare python session (non-jupyter). Not sure it would help, but curious. – Oliver Jun 22 '21 at 12:29
  • The platform is quite heavily monitored. I'm not sure that trying to get covert admin rights would enhance my continued employment prospects! It does crash a bare python session though (e.g. in Spyder) - see my most recent update - so I guess it's not jupyter/venv specific. – majr Jun 22 '21 at 12:38

2 Answers2

1

I have fixed this by adding system environment variables: R_HOME, R_USER, and Path. [windows10, R 4.1.0, rpy2 3.4.5, python 3.8]

cslhb
  • 11
  • 1
  • Thanks @cslhb, I ran out of time to solve this and reluctantly refactored to a non-rpy2 solution. I'd really like to get it working for other projects though. Could you say a bit more? How did you determine what the right values were, and where did you add them? To a venv, or globally? – majr Jul 14 '21 at 08:25
  • What are the values of that variables? – buhtz Jul 27 '22 at 08:08
0

For me, this problem was caused by a .Rprofile file (a start-up file for R) that was located at %USERPROFILE%\Documents\.Rprofile (on Windows 11) which contained the following R code:

if (interactive()) {
  require(conflicted)
}

The initialization of the R interpreter (as done through rpy2) was not able to handle loading the conflicted library. To understand why I created that .Rprofile, please review this tidyverse blog post: https://www.tidyverse.org/blog/2018/06/conflicted/

Commenting out those three lines in the .Rprofile fixed the import rpy2.robjects crash for me.

geltz
  • 1