23

I'm writing an R package which depends upon many other packages. When I load too many packages into the session I frequently got this error:

Error in dyn.load(file, DLLpath = DLLpath, ...) : 
  unable to load shared object '/Library/Frameworks/R.framework/Versions/3.2/Resources/library/proxy/libs/proxy.so':
  `maximal number of DLLs reached...

This post Exceeded maximum number of DLLs in R pointed out that the issue is with the Rdynload.c of the base R code: #define MAX_NUM_DLLS 100

Is there any way to bypass this issue except modifying and building from source?

zx8754
  • 52,746
  • 12
  • 114
  • 209
Qin Zhu
  • 331
  • 1
  • 2
  • 11
  • Have you tried to use Microsoft R? I'm not sure that'll work, but it may be a viable alternative. – hrbrmstr May 02 '16 at 03:33
  • I just checked the source code of Microsoft R. I don't think they changed the '#define MAX_NUM_DLLS 100' code. Also some packages which require compilation cannot be installed. – Qin Zhu May 02 '16 at 18:10

4 Answers4

19

As of R 3.4, you can set a different max number of DLLs using and environmental variable R_MAX_NUM_DLLS. From the release notes:

The maximum number of DLLs that can be loaded into R e.g. via dyn.load() can now be increased by setting the environment variable R_MAX_NUM_DLLS before starting R.

Stuart R. Jefferys
  • 943
  • 1
  • 13
  • 23
  • To set this globally, should it be added to Renviron or Renviron.site? – FM Kerckhof May 04 '17 at 11:04
  • See `?Startup`, or google gives multiple references describing startup configuration for R, e.g. [Understanding R's startup](https://rviews.rstudio.com/2017/04/19/r-for-enterprise-understanding-r-s-startup/) – Stuart R. Jefferys May 05 '17 at 20:39
  • from 'efficient R programming': The .Renviron file is used to store system variables. It follows a similar start up routine to the .Rprofile file: R first looks for a global .Renviron file, then for local versions. A typical use of the .Renviron file is to specify the R_LIBS path – Garini Mar 19 '18 at 10:28
  • On Windows there is an error when loading R is this environment variable is set to a value larger than 1000. –  Mar 23 '22 at 16:22
16

Increasing that number is of course "possible"... but it also costs a bit (adding to the fixed memory footprint of R).

I did not set that limit, but I'm pretty sure it was also meant as reminder for the useR to "clean up" a bit in her / his R session, i.e., not load package namespaces unnecessarily. I cannot yet imagine that you need > 100 packages | namespaces loaded in your R session. OTOH, some packages nowadays have a host of dependencies, so I agree that this at least may happen accidentally more frequently than in the past.

The real solution of course would be a code improvement that starts with a relatively small number of "DLLinfo" structures (say 32), and then allocates more batches (of size say 32) if needed.

Patches to the R sources (development trunk in subversion at https://svn.r-project.org/R/trunk/ ) are very welcome!

---- added Jan.26, 2017: In the mean time, we've had a public bug report about this, a proposed patch (which was not good enough: There is always an OS dependent limit on the number of open files), and today that bug report has been closed by R core member @TomasKalibera who implemented new code where the maximal number of loaded DLLs is set at

pmax(100, pmin(1000, 0.6* OS_dependent_getrlimit_or_equivalent()))

and so on Windows and Linux (and not yet tested, but "almost surely" macOS), the limit should be considerably higher than previously.

----- Update #2 (written Jan.5, 2018):
In Oct'17, the above change was made more automatic with the following commit to the sources (of the development version of R - only!)

r73545 | kalibera | 2017-10-12 14:41:20

Increase the number of DLLs that can be loaded by default. If needed, increase the soft limit on open files.

and on the help page ?dyn.load (https://stat.ethz.ch/R-manual/R-devel/library/base/html/dynload.html) the ulimit -n <num_open_files> is now mentioned (section Note close to bottom).

So you might consider using R's development version till that becomes "main stream" in April.
Alternatively, you do (in a terminal / shell)

ulimit -n 2048

and then start R from that terminal. Tomas Kalibera mentioned this to work on macOS.

Martin Mächler
  • 4,619
  • 27
  • 27
  • would the R Core team be open to bumping MAX_NUM_DLLS to 200 or so? Or is there more than meets the eye here? I'm working on the R package mlr and we run into this problem when performing unit tests for all the learners – Steve Bronder Dec 19 '16 at 23:46
  • Yes, there's slightly more.. see the threads on the R-devel mailing list, see e.g. my recent post https://stat.ethz.ch/pipermail/r-devel/2016-December/073529.html (and messages around it). I hope we have something by R 3.4.0 ca April'17 (and quite a bit sooner if you are willing to work with the development version of R. – Martin Mächler Dec 24 '16 at 13:04
  • Have the same problem here - was coding a package with a lot of dependencies, and I am now also getting this error... – Tom Wenseleers Jan 25 '17 at 18:25
  • @MartinMächler: was this then added to R 3.4.0? – FM Kerckhof May 04 '17 at 11:04
  • Yes, it was added to R 3.4.0 --- however not quite as above. It was found / that the above was still "too dangerous" / not okay in some cases, namely when the number of open files is limited (and it typically is limited quite a bit on (some?) Linux (such as my Fedora 24) and so R 3.4.0 now takes the default from an environment variable `R_MAX_NUM_DLLS` (which on Linux cannot be larger than 1000). E.g. on my current (quite strong) desktop: – Martin Mächler May 15 '17 at 16:02
  • @MartinMächler: Thanks for your answer. It seems in MacOS system the maximum DLL number can only reach 153 otherwise R will report fatal error. Do you know if there is anyway to bypass this? Nowadays bioconductor packages generally have many dependences and could easily exhaust this limit. – Qin Zhu Jan 03 '18 at 18:54
  • @MartinMächler: Thanks for the update! That's awesome! – Qin Zhu Jan 06 '18 at 18:10
5

I had this issue with the simpleSingleCell library in bioconductor

On the macOS you can't exceed 256. So I set my .Renviron in my home dir R_MAX_NUM_DLLS=150

steve
  • 51
  • 2
  • 2
-2

It's easy Go to the environment variable and edit 

variable_name = R_MAX_NUM_DLL
value = 1000

Restart R worked well for me

Garini
  • 1,088
  • 16
  • 29
xtramous
  • 1
  • 1