1

I am attempting to import an R package from a specific R path in a conda environment, but I keep receiving the following error:

Error: package or namespace load failed for ‘marginaleffects’ in library.dynam(lib, package, package.lib):
shared object ‘data_table.so’ not found

I am struggling to understand not just how to fix this error, but also what this error means or where it is coming from.

Specifically, I am deploying a web application with the Python-based streamlit framework and am using the package rpy2 to interface from an existing R codebase to Python. I use conda/mamba as the package manager on my machine. I have installed all R packages using their conda-forge distribution as possible. There are several R packages that are not available, so I have installed them in R to the path of my desired library with with: install.packages("marginaleffects", dep = TRUE, lib = "/Users/myuser/mambaforge/envs/myenv/lib/R/library", verbose = TRUE). The package then appears to exist in the directory .../myenv/lib/R/library, in installed.packages(), and in RStudio's list of packages.

However, when I attempt to load the package with library(marginaleffects, lib.loc="/Users/myuser/mambaforge/envs/myenv/lib/R/library") I receive the error noted above. Note that if I import this Packages in an R code run in the streamlit app, it raises a slightly different error relating to a .dylib shared object, instead of an '.so` shared object.

RRuntimeError: Error: package or namespace load failed for ‘marginaleffects’ in library.dynam(lib, package, package.lib): shared object ‘marginaleffects.dylib’ not found

I have the vaguest sense this has something to do with a compiler error. I have also been unsuccessful in installing the package through the rpy2 framework, receiving a long and fairly inscrutable error message.

EDIT

The code I have run so far for reproducibility on a MacOS (Ventura, same issue happened on Big Sur):

zsh> mamba create -n myenv r-essentials r-base python=3.8
zsh> mamba activate myenv
zsh> mamba install -c r rpy2 
zsh> mamba install -c conda-forge pandas
zsh> mamba install -c conda-forge streamlit

R> install.packages("clarify", dep = TRUE, lib = "/Users/myname/mambaforge/envs/myenv/lib/R/library", verbose = TRUE)
R> install.packages("marginaleffects", dep = TRUE, lib = "/Users/myname/mambaforge/envs/myenv/lib/R/library", verbose = TRUE)
R> library(marginaleffects, lib.loc="/Users/myuser/mambaforge/envs/myenv/lib/R/library")

Error: package or namespace load failed for ‘marginaleffects’ in library.dynam(lib, package, package.lib):
shared object ‘data_table.so’ not found

EDIT2

I am running osx-64.

stat_is_quo
  • 53
  • 5
  • [https://stackoverflow.com/questions/42571407/r-package-installation-fails-cant-find-existing-dylib](Other question) could be usefull info. – user12256545 Jul 01 '23 at 18:23
  • 1
    This is too vague. Please create a minimal example that others can recreate the issue with. Typically, [DLL issues can arise from mixing channels](https://conda-forge.org/docs/user/tipsandtricks.html#using-multiple-channels), but it's not clear that applies here. Is everything installed with Conda? or was `install.packages` used? (it generally shouldn't be) – merv Jul 02 '23 at 15:49
  • Thanks @merv. I have added the code run so far to reproduce the issue. If not install.packages, how would you add a package to a conda/mamba environment when it does not have a conda distribution? – stat_is_quo Jul 05 '23 at 15:45
  • 1
    I have directions in [this answer](https://stackoverflow.com/a/69921221/570918). I'm on the Conda Forge R team, so if you ever have hiccups getting something added there, just ping me (**@mfansler** on GH). Definitely `marginaleffects` is available already (`r-marginaleffects`) and I'll add `clarify` for you (`r-clarify`). Note that CRAN packages are always prefixed with `r-` and converted to lowercase. – merv Jul 05 '23 at 20:22
  • Many thanks @merv! The `r-clarify` package was available for download. Unfortunately, I receive the message `Could not solve for environment specs The following package could not be installed` with a package tree. When I attempt to install the basal package in the tree, I get the same error with `The following package could not be installed └─ llvm-tools 13.0.1** is requested and can be installed`. Is this (potentially) related to the `r-clarify` build or is this likely a deeper issue? (I'm not seeing an obvious answer online for what this error means.) – stat_is_quo Jul 05 '23 at 23:19
  • 1
    You should not be using the `r` channel - only `conda-forge`. However, that still ends in error, so I gave an answer. Basically, still need `r-mvnfast` to be built for **osx-arm64**. Such migrations are highly variable in turnaround time, but I'll try to swing back with updates. – merv Jul 06 '23 at 14:28

2 Answers2

1

I can recreate the issue on osx-arm64 platform (note that everything works fine for osx-64):

so-rpy2-clarify.yaml

name: so-rpy2-clarify
channels:
  - conda-forge  # only Conda Forge
  - nodefaults   # ignore user config
dependencies:
  ## Python
  - python=3.8
  - rpy2
  - pandas
  - streamlit

  ## R
  - r-base=4.3  # always define R version
  - r-clarify
  - r-essentials
  - r-marginaleffects

which running with:

mamba env create -f so-rpy2-clarify.yaml

gives:

Could not solve for environment specs
The following package could not be installed
└─ r-clarify is not installable because it requires
   └─ r-mvnfast, which does not exist (perhaps a missing channel).

This package isn't yet being built for osx-arm64, but we can put in a request (see the documentation on "Apple Silicon Builds").

Let's also check to make sure nothing else also is needed, since the conflict reporting usually just reports the first problem. We use mamba repoquery for this:

mamba repoquery depends -c conda-forge -p osx-arm64 r-clarify

Executing the query r-clarify

conda-forge/osx-arm64                                       Using cache
conda-forge/noarch                                          Using cache


 Name                        Version Build             Channel    
───────────────────────────────────────────────────────────────────
 r-clarify                   0.1.3   r42hc72bb7e_0     conda-forge
 r-rlang                     0.4.11  r40h6f62c66_0     conda-forge
 r-ggplot2                   3.1.0   r351h6115d3f_1000 conda-forge
 r-pbapply                   1.3_4   r351h6115d3f_1000 conda-forge
 r-mvnfast >>> NOT FOUND <<<                                      
 r-base                      4.2.3   heabe65b_0        conda-forge
 r-chk                       0.9.0   r43hc72bb7e_1     conda-forge
 r-insight                   0.19.3  r43hc72bb7e_0     conda-forge
 r-marginaleffects           0.9.0   r42hc72bb7e_0     conda-forge

Looks like r-mvnfast is the only missing dependency.

merv
  • 67,214
  • 13
  • 180
  • 245
  • 1
    Huge thanks @merv! I just tried this and it worked for me. I'll just add that I was having the problem posted above even with osx-64 (which I will edit my post to specify). I think for me the fix may have been something as simple as specifying the R version or avoiding defaults. – stat_is_quo Jul 06 '23 at 14:33
0

In addition to @merv's answer above, which I have accepted, I was able to resolve this issue with a slightly different approach by doing two things. This may be helpful for anyone troubleshooting a similar problem.

  1. Sequencing: Installing streamlit last and using pip instead of conda. Once I had installed all the other required packages, I could not get conda to find a compatible distribution of streamlit. Installing with pip seems to work fine, however.
  2. Consistency in installation: Installing all R packages through conda (technically mamba) instead of install.packages in R. (Thanks again to @merv for rapidly getting r-clarify up.)
stat_is_quo
  • 53
  • 5