Heroku RPy RHOME discovery

Question

I am trying to run a multipack Heroku app that uses Python and R. I have the multi buildpack installing R first, but RPy's installation is not able to find R despite my modifications to $PATH. What is going on here? R sees RHOME as "/app/vendor/R/lib64/R" when init.r runs.

-----> Fetching custom git buildpack... done
-----> Multipack app detected
=====> Downloading Buildpack: https://github.com/virtualstaticvoid/heroku-buildpack-r
=====> Detected Framework: R
   Vendoring R 2.15.1
   Downloading and unpacking R binaries
   Executing init.r script
[1] "/app/vendor/R/lib64/R" #This is me dumping out RHOME from init.r
   R 2.15.1 successfully installed
=====> Downloading Buildpack: https://github.com/heroku/heroku-buildpack-python
=====> Detected Framework: Python
-----> No runtime.txt provided; assuming python-2.7.4.
-----> Using Python runtime (python-2.7.4)
-----> Installing dependencies using Pip (1.3.1)
   Downloading/unpacking rpy2==2.3.5 (from -r requirements/base.txt (line 22))
     Running setup.py egg_info for package rpy2

       sh: R: not found
       Error: Tried to guess R's HOME but no R command in the PATH.
       Complete output from command python setup.py egg_info:
       running egg_info

   creating pip-egg-info/rpy2.egg-info

   writing pip-egg-info/rpy2.egg-info/PKG-INFO

   writing top-level names to pip-egg-info/rpy2.egg-info/top_level.txt

   writing dependency_links to pip-egg-info/rpy2.egg-info/dependency_links.txt

   writing manifest file 'pip-egg-info/rpy2.egg-info/SOURCES.txt'

   warning: manifest_maker: standard file '-c' not found



   sh: R: not found

   Error: Tried to guess R's HOME but no R command in the PATH.

   ----------------------------------------
   Command python setup.py egg_info failed with error code 1 in /tmp/pip-build-u32629/rpy2
   Storing complete log in /app/.pip/pip.log
 !     Heroku push rejected, failed to compile Multipack app

To git@heroku.com:D.git
 ! [remote rejected] master -> master (pre-receive hook declined)

(venvddd)ben@Watt:~/Projects/D/D$ heroku config:get PATH /home/ben/Projects/D/venvddd/bin:/usr/local/heroku/bin:/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/app/vendor/R/lib64/R:/app/vendor/R/lib64/R/bin

rpy2's `setup.py` is using `os.popen("R RHOME")`. May be "something" is happening when the child subprocess is created on Heroku. Since that function is deprecated in Python, it would be worth trying whether the `subprocess` module solves the problem. — lgautier, May 09 '13 at 19:36
Turns out the issue is buildpack installation being transactional on heroku, so $PATH is changing between installations. Forking the heroku python buildpack and tweaking the $PATH variable makes installation run, but I am hitting newer and more interesting import errors later on. More as it develops. — bwarren2, May 13 '13 at 18:11
Got it done! This is much trickier than you might originally expect, with hacking in several custom buildpacks and configuration of environment variables to manage imports. The secret is that Heroku buildpacks are transactional, though. — bwarren2, May 18 '13 at 00:25
If you see where the rpy2 install process could be made better, do not hesitate to make suggestions (or contribute patches). — lgautier, May 18 '13 at 12:27

score 1 · Accepted Answer · edited May 23 '17 at 12:14

After some helpful suggestions from @bwarren2, I think I have a somewhat cleaner way to use R, python, and rpy2 together on Heroku.

In this example, I'm using the python buildpack heroku-buildpack-python-sklearn, that contains numpy, scipy, and scikit-learn from binary builds. The Rpy2 library has nice numpy integration so you are likely to want to start with this. If not, then the same approach works with normal python buildpack.

Make a .buildpacks file like this:

https://github.com/virtualstaticvoid/heroku-buildpack-r.git
https://github.com/dbrgn/heroku-buildpack-python-sklearn/

An optional init.r file that installs R libraries, and this requirements.txt file:

numpy==1.7.0
scipy==0.11.0
scikit-learn==0.13.1
matplotlib==1.1.0
rpy2==2.3.8

Because we're picking up binary builds of these (due to BLAS and other dependencies), the version numbers must exactly match.

Then we do the normal process to use the multi-build packs. However, the python buildpack needs to know where R and some libraries were installed. The slug building system on Heroku doesn't pass all the environment variables set in the R buildpack to the python buildpack.

However, we can use the new user-env-compile feature from Heroku labs, and set variables for PATH and LD_LIBRARY_PATH explicitly. Here's what I did...

# make a test repo
git init 
# add our files
git add init.r
git add requirements.txt
git add .buildpacks
# commit the files
git ci -m"testing using user-env-compile"

# create a new app using the multi buildpack code
heroku create --buildpack https://github.com/ddollar/heroku-buildpack-multi.git

# turn on user-env-compile, that allows config vars when compiling slug
# see https://devcenter.heroku.com/articles/labs-user-env-compile
heroku labs:enable user-env-compile

# set the path variables explicitly, so python knows where R is located
heroku config:set PATH=/app/vendor/R/bin:/app/vendor/gcc-4.3/bin:/app/.heroku/python/bin:/usr/local/bin:/usr/bin:/bin
heroku config:set LD_LIBRARY_PATH=/usr/lib:/usr/local/lib:/app/vendor/R/lib64/R/modules:/app/vendor/R/lib64/R/lib:/app/vendor/gcc-4.3/lib64


# create the slug
git push heroku master

# we can now access R from python using rpy2
$ heroku run python
Running `python` attached to terminal... up, run.1144
Python 2.7.4 (default, Apr  6 2013, 22:14:13) 
[GCC 4.4.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import rpy2.robjects as robjects
>>> pi = robjects.r['pi']
>>> pi[0] 
3.141592653589793

Edits: I updated LD_LIBRARY_PATH to contain /usr/lib and /usr/local/lib, to avoid the problem described here.

Also added matplotlib==1.1.0 to requirements.txt, which is an older version as described here.

In my case, I have to add these two paths into LD_LIBRARY_PATH: /app/vendor/.apt/usr/lib/libblas/:/app/vendor/.apt/usr/lib/ — ruhanbidart, May 01 '15 at 16:04

Heroku RPy RHOME discovery

1 Answers1

Linked