How to use virtualenv system-site-packages on AWS Elastic Beanstalk?

Question

I have to work on an existing python (django) application that runs on AWS Elastic Beanstalk. It seems that all the requirements (from requirements.txt) are installed in a virtual env (I hope I'm right on that). My problem is that I have installed some libraries using yum. My dependencies config for my beanstalk instance looks like this:

packages:
 yum:
  python27-devel: []
  git: []
  nginx: []
  pcre-devel: []
  freetype-devel: []
  libpng-devel: []
  postgresql93-devel: []
  graphviz-devel: []
  blas-devel: []
  atlas-devel: []
  lapack-devel: []
  gcc-c++: []
  python27-numpy: []
  python27-matplotlib: []
  python27-psycopg2: []

As you can see I explicitly install some python libraries like numpy, matplotlib and psycopg2 but since my application seems to run in a virtual env these libraries are not accessible for my app so it crashes when launching. I got this type of errors:

Command failed on instance. Return code: 1 Output: (TRUNCATED)....7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 20, in raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e) django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: No module named psycopg2. container_command 01_migrate in .ebextensions/02-python.config failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.

I've read that you can allow a virtual env to access the system libraries with the option system-site-packages but how to do so for a AWS elastic beanstalk instance?

EDIT: What I don't want to do is installing numpy, matplotlib and psycopg2 with pip since it compiles everything and that's very long. It is why I want to install these libraries with yum, to get a precompiled version. And I want these installations to be automatic, I don't want to ssh on the machine to install things manually. This is where my problem appears. :)

Thanks a lot for your time!

(Sorry for my english, it's not my native language)

score 3 · Answer 1 · edited May 23 '17 at 12:24

TL;DR: Add the following command to a config file under .ebextensions folder:

commands:
  01_add_dist_files_to_site:
    command: "echo /usr/lib64/python2.7/dist-packages > /opt/python/run/venv/local/lib64/python2.7/site-packages/dist.pth"

This should expose the libraries under /usr/lib64/python2.7/dist-packages (where yum installs numpy and scipy for example...) to the interpreter used by EB's virtualenv.

Elastic Beanstalk is using /opt/python/run/venv/bin/pip install -r requirements.txt to install your requirements. You need to set the python virtualenv in a way that will make it exposed to your installed packages.

Examining the site libraries that are accessible to EB's venv I found:

(venv)[root@ip-XXXXXX app]# /opt/python/run/venv/bin/python2.7 -msite
sys.path = [
    '',
    '',
    '/opt/python/run/venv/local/lib64/python2.7/site-packages',
    '/opt/python/run/venv/local/lib/python2.7/site-packages',
    '/opt/python/run/venv/lib64/python2.7',
    '/opt/python/run/venv/lib/python2.7',
    '/opt/python/run/venv/lib64/python2.7/site-packages',
    '/opt/python/run/venv/lib/python2.7/site-packages',
    '/opt/python/run/venv/lib64/python2.7/lib-dynload',
    '/usr/lib64/python2.7',
    '/usr/lib/python2.7',
]

So it is enough to add a .pth file which contains paths to your desired libraries, to one of the listed folders (see here). This can be done as mentioned at the beginning. Note that you need your requirements.txt to hold python packages with the exact versions of the ones installed using yum, in order to skip unnecessary installation steps.

How to use virtualenv system-site-packages on AWS Elastic Beanstalk?

1 Answers1