Configuring compilers on Apple silicon (Big Sur, Monterey, Ventura) for Rcpp and other tools

Question

I'm trying to use packages that require Rcpp in R on my M1 Mac, which I was never able to get up and running after purchasing this computer. I updated it to Monterey in the hope that this would fix some installation issues but it hasn't. I tried running the Rcpp check from this page but I get the following error:

> Rcpp::sourceCpp("~/github/helloworld.cpp")

ld: warning: directory not found for option '-L/opt/R/arm64/gfortran/lib/gcc/aarch64-apple-darwin20.2.0/11.0.0'
ld: warning: directory not found for option '-L/opt/R/arm64/gfortran/lib'
ld: library not found for -lgfortran
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [sourceCpp_4.so] Error 1
clang++ -arch arm64 -std=gnu++14 -I"/Library/Frameworks/R.framework/Resources/include" -DNDEBUG -I../inst/include   -I"/Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/library/Rcpp/include" -I"/Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/library/RcppArmadillo/include" -I"/Users/afredston/github" -I/opt/R/arm64/include   -fPIC  -falign-functions=64 -Wall -g -O2  -c helloworld.cpp -o helloworld.o
clang++ -arch arm64 -std=gnu++14 -dynamiclib -Wl,-headerpad_max_install_names -undefined dynamic_lookup -single_module -multiply_defined suppress -L/Library/Frameworks/R.framework/Resources/lib -L/opt/R/arm64/lib -o sourceCpp_4.so helloworld.o -L/Library/Frameworks/R.framework/Resources/lib -lRlapack -L/Library/Frameworks/R.framework/Resources/lib -lRblas -L/opt/R/arm64/gfortran/lib/gcc/aarch64-apple-darwin20.2.0/11.0.0 -L/opt/R/arm64/gfortran/lib -lgfortran -lemutls_w -lm -F/Library/Frameworks/R.framework/.. -framework R -Wl,-framework -Wl,CoreFoundation
Error in Rcpp::sourceCpp("~/github/helloworld.cpp") : 
  Error 1 occurred building shared library.

I get that it can't "find" gfortran. I installed this release of gfortran for Monterey. When I type which gfortran into Terminal, it returns /opt/homebrew/bin/gfortran. (Maybe this version of gfortran requires Xcode tools that are too new—it says something about 13.2 and when I run clang --version it says 13.0—but I don't see another release of gfortran for Monterey?)

I also appended /opt/homebrew/bin: to PATH in R so it looks like this now:

> Sys.getenv("PATH")

[1] "/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/Library/TeX/texbin:/Applications/RStudio.app/Contents/MacOS/postback"

Other things I checked:

• Xcode command line tools is installed (which clang returns /usr/bin/clang).
• Files ~/.R/Makevars and ~/.Renviron don't exist.

Here's my session info:

R version 4.1.1 (2021-08-10)
Platform: aarch64-apple-darwin20 (64-bit)
Running under: macOS Monterey 12.1

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.1-arm64/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_4.1.1           tools_4.1.1              RcppArmadillo_0.10.7.5.0
[4] Rcpp_1.0.7        

Answer 1

Background

Currently (2023-04-14), CRAN builds R 4.3.x binaries for Apple silicon using Apple Clang from Command Line Tools for Xcode 14.2 and using an experimental fork of GNU Fortran 12.

If you obtain R from CRAN (i.e., here), then you need to replicate CRAN's compiler setup on your system before building R packages that contain C/C++/Fortran code from their sources (and before using Rcpp, etc.). This requirement ensures that your package builds are compatible with R itself.

A further complication is the fact that Apple Clang doesn't support OpenMP, so you need to do even more work to compile programs that make use of multithreading. You could circumvent the issue by building R itself, all R packages, and all external libraries from sources with LLVM Clang, which does support OpenMP, but that approach is onerous and "for experts only".

There is another approach that has been tested by a few people, including Simon Urbanek, the maintainer of R for macOS. It is experimental and also "for experts only", but it works on my machine and is much simpler than learning to build R and other libraries yourself.

Instructions for obtaining a working toolchain

Warning: These come with no warranty and could break at any time. Some level of familiarity with C/C++/Fortran program compilation, Makefile syntax, and Unix shells is assumed. Everyone is encouraged to consult official documentation, which is more likely to be maintained than answers on SO. As usual, sudo at your own risk.

I will try to address compilers and OpenMP support at the same time. I am going to assume that you are starting from nothing. Feel free to skip steps you've already taken, though you might find a fresh start helpful.

I've tested these instructions on a machine running Ventura, but they should also work on Big Sur and Monterey.

1. Download an R 4.3.x binary from CRAN here and install. Be sure to select the binary built for Apple silicon.

2. Run

   $ sudo xcode-select --install
   

   in Terminal to install the latest release version of Apple's Command Line Tools for Xcode (well, the latest one supporting your macOS version), which includes Apple Clang. You can obtain older versions from your browser here. CRAN builds R 4.3.x for macOS 11.0 and newer using Xcode 14.2.

3. Download the GNU Fortran binary provided here and install by unpacking to root:

   $ curl -LO https://github.com/R-macos/gcc-12-branch/releases/download/12.2-darwin-r0/gfortran-12.2-darwin20-r0-universal.tar.xz
   $ sudo tar xvf gfortran-12.2-darwin20-r0-universal.tar.xz -C /
   $ sudo ln -sfn $(xcrun --show-sdk-path) /opt/gfortran/SDK
   

   The last command updates a symlink inside of the installation so that it points to an SDK inside of your Command Line Tools installation.

4. Download an OpenMP runtime suitable for your Apple Clang version here and install by unpacking to root. You can query your Apple Clang version with clang --version. For example, I have version 1403.0.22.14.1, so I did:

   $ curl -LO https://mac.r-project.org/openmp/openmp-14.0.6-darwin20-Release.tar.gz
   $ sudo tar xvf openmp-14.0.6-darwin20-Release.tar.gz -C /
   

   After unpacking, you should find these files on your system:

   /usr/local/lib/libomp.dylib
   /usr/local/include/ompt.h
   /usr/local/include/omp.h
   /usr/local/include/omp-tools.h
   

5. Add the following lines to $(HOME)/.R/Makevars, creating the file if necessary.

   CPPFLAGS += -I/usr/local/include -Xclang -fopenmp
   LDFLAGS += -L/usr/local/lib -lomp
   

6. Test that you are able to use R to compile a C or C++ program with OpenMP support while linking relevant libraries from the GNU Fortran installation (indicated by the -l flags in the output of R CMD CONFIG FLIBS).

   The most transparent approach is to use R CMD SHLIB directly. In a temporary directory, create an empty source file omp_test.c and add the following lines:

   #ifdef _OPENMP
   # include <omp.h>
   #endif
   
   #include <Rinternals.h>
   
   SEXP omp_test(void)
   {
   #ifdef _OPENMP
       Rprintf("OpenMP threads available: %d\n", omp_get_max_threads());
   #else
       Rprintf("OpenMP not supported\n");
   #endif
       return R_NilValue;
   }
   

   Compile it:

   $ R CMD SHLIB omp_test.c $(R CMD CONFIG FLIBS)
   

   Then call the compiled C function from R:

   $ R -e 'dyn.load("omp_test.so"); invisible(.Call("omp_test"))'
   

   OpenMP threads available: 8
   

   If the compiler or linker throws an error, or if you find that OpenMP is still not supported, then one of us has made a mistake. Please report any issues.

   Note that you can implement the same test using Rcpp, if you don't mind installing it:

   library(Rcpp)
   registerPlugin("flibs", Rcpp.plugin.maker(libs = "$(FLIBS)"))
   sourceCpp(code = '
   #ifdef _OPENMP
   # include <omp.h>
   #endif
   
   #include <Rcpp.h>
   
   // [[Rcpp::plugins(flibs)]]
   // [[Rcpp::export]]
   void omp_test()
   {
   #ifdef _OPENMP
       Rprintf("OpenMP threads available: %d\\n", omp_get_max_threads());
   #else
       Rprintf("OpenMP not supported\\n");
   #endif
       return;
   }
   ')
   omp_test()
   

   OpenMP threads available: 8
   

References

Everything is a bit scattered:

• R Installation and Administration manual [link]
• Writing R Extensions manual [link]
• R for macOS Developers web page [link]

Answer 2

I resolved this issue by adding a path to the homebrew installation of gfortran to my ~/.R/Makevars following these instructions: https://pat-s.me/transitioning-from-x86-to-arm64-on-macos-experiences-of-an-r-user/#gfortran

Answer 3

Tested the following process for making multithread data.table work in a M2 MacBook Pro (macOS Monterey)

Steps are mostly the same with this answer by the user inferator.

1. Download and install R from CRAN
2. Download and install RStudio with developer tools
3. Run the following commands in terminal to install OpenMP

curl -O https://mac.r-project.org/openmp/openmp-12.0.1-darwin20-Release.tar.gz
sudo tar fvxz openmp-12.0.1-darwin20-Release.tar.gz -C /

4. Add compiler flags to connect clan w/ OpenMP. In terminal, write the following:

cd ~
mkdir .R
nano .R/Makevars

• Inside the opened Makevars file paste the following lines. Once finished, hit command+O and then Enter to save. Do a command+X to close the editor.

CPPFLAGS += -Xclang -fopenmp
LDFLAGS += -lomp

5. Download and run the installer for gfortran by downloading gfortran-ARM-12.1-Monterey.dmg from the respective GitHub repo

This concludes the steps regarding enabling OpenMP and (hopefully) Rcpp in R under a M2 chip system.

Now, for testing that everything works with data.table I did the following

Open RStudio and run

install.packages("data.table", type = "source")

If everything is done correctly, the package should compile without any errors and return the following when running getDTthreads(verbose = TRUE):

  OpenMP version (_OPENMP)       201811
  omp_get_num_procs()            8
  R_DATATABLE_NUM_PROCS_PERCENT  unset (default 50)
  R_DATATABLE_NUM_THREADS        unset
  R_DATATABLE_THROTTLE           unset (default 1024)
  omp_get_thread_limit()         2147483647
  omp_get_max_threads()          8
  OMP_THREAD_LIMIT               unset
  OMP_NUM_THREADS                unset
  RestoreAfterFork               true
  data.table is using 4 threads with throttle==1024. See ?setDTthreads.
[1] 4