Using moviepy, scipy and numpy in amazon lambda

Question

I'd like to generate video using AWS Lambda feature.

I've followed instructions found here and here.

And I now have the following process to build my Lambda function:

Step 1

Fire a Amazon Linux EC2 instance and run this as root on it:

#! /usr/bin/env bash

# Install the SciPy stack on Amazon Linux and prepare it for AWS Lambda

yum -y update
yum -y groupinstall "Development Tools"
yum -y install blas --enablerepo=epel
yum -y install lapack --enablerepo=epel
yum -y install atlas-sse3-devel --enablerepo=epel
yum -y install Cython --enablerepo=epel
yum -y install python27
yum -y install python27-numpy.x86_64
yum -y install python27-numpy-f2py.x86_64
yum -y install python27-scipy.x86_64

/usr/local/bin/pip install --upgrade pip
mkdir -p /home/ec2-user/stack
/usr/local/bin/pip install moviepy -t /home/ec2-user/stack

cp -R /usr/lib64/python2.7/dist-packages/numpy /home/ec2-user/stack/numpy
cp -R /usr/lib64/python2.7/dist-packages/scipy /home/ec2-user/stack/scipy

tar -czvf stack.tgz /home/ec2-user/stack/*

Step 2

I scp the resulting tarball to my laptop. And then run this script to build a zip archive.

#! /usr/bin/env bash

mkdir tmp
rm lambda.zip
tar -xzf stack.tgz -C tmp

zip -9 lambda.zip process_movie.py
zip -r9 lambda.zip *.ttf
cd tmp/home/ec2-user/stack/
zip -r9 ../../../../lambda.zip *

process_movie.py script is at the moment only a test to see if the stack is ok:

def make_movie(event, context):
    import os
    print(os.listdir('.'))
    print(os.listdir('numpy'))
    try:
        import scipy
    except ImportError:
        print('can not import scipy')

    try:
        import numpy
    except ImportError:
        print('can not import numpy')

    try:
        import moviepy
    except ImportError:
        print('can not import moviepy')

Step 3

Then I upload the resulting archive to S3 to be the source of my lambda function. When I test the function I get the following callstack:

START RequestId: 36c62b93-b94f-11e5-9da7-83f24fc4b7ca Version: $LATEST
['tqdm', 'imageio-1.4.egg-info', 'decorator.pyc', 'process_movie.py', 'decorator-4.0.6.dist-info', 'imageio', 'moviepy', 'tqdm-3.4.0.dist-info', 'scipy', 'numpy', 'OpenSans-Regular.ttf', 'decorator.py', 'moviepy-0.2.2.11.egg-info']
['add_newdocs.pyo', 'numarray', '__init__.py', '__config__.pyc', '_import_tools.py', 'setup.pyo', '_import_tools.pyc', 'doc', 'setupscons.py', '__init__.pyc', 'setup.py', 'version.py', 'add_newdocs.py', 'random', 'dual.pyo', 'version.pyo', 'ctypeslib.pyc', 'version.pyc', 'testing', 'dual.pyc', 'polynomial', '__config__.pyo', 'f2py', 'core', 'linalg', 'distutils', 'matlib.pyo', 'tests', 'matlib.pyc', 'setupscons.pyc', 'setup.pyc', 'ctypeslib.py', 'numpy', '__config__.py', 'matrixlib', 'dual.py', 'lib', 'ma', '_import_tools.pyo', 'ctypeslib.pyo', 'add_newdocs.pyc', 'fft', 'matlib.py', 'setupscons.pyo', '__init__.pyo', 'oldnumeric', 'compat']
can not import scipy
'module' object has no attribute 'core': AttributeError
Traceback (most recent call last):
  File "/var/task/process_movie.py", line 91, in make_movie
    import numpy
  File "/var/task/numpy/__init__.py", line 122, in <module>
    from numpy.__config__ import show as show_config
  File "/var/task/numpy/numpy/__init__.py", line 137, in <module>
    import add_newdocs
  File "/var/task/numpy/numpy/add_newdocs.py", line 9, in <module>
    from numpy.lib import add_newdoc
  File "/var/task/numpy/lib/__init__.py", line 13, in <module>
    from polynomial import *
  File "/var/task/numpy/lib/polynomial.py", line 11, in <module>
    import numpy.core.numeric as NX
AttributeError: 'module' object has no attribute 'core'

END RequestId: 36c62b93-b94f-11e5-9da7-83f24fc4b7ca
REPORT RequestId: 36c62b93-b94f-11e5-9da7-83f24fc4b7ca  Duration: 112.49 ms Billed Duration: 200 ms     Memory Size: 1536 MB    Max Memory Used: 14 MB

I cant understand why python does not found the core directory that is present in the folder structure.

EDIT:

Following @jarmod advice I've reduced the lambdafunction to:

def make_movie(event, context):
    print('running make movie')
    import numpy

I now have the following error:

START RequestId: 6abd7ef6-b9de-11e5-8aee-918ac0a06113 Version: $LATEST
running make movie
Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python intepreter from there.: ImportError
Traceback (most recent call last):
  File "/var/task/process_movie.py", line 3, in make_movie
    import numpy
  File "/var/task/numpy/__init__.py", line 127, in <module>
    raise ImportError(msg)
ImportError: Error importing numpy: you should not try to import numpy from
        its source directory; please exit the numpy source tree, and relaunch
        your python intepreter from there.

END RequestId: 6abd7ef6-b9de-11e5-8aee-918ac0a06113
REPORT RequestId: 6abd7ef6-b9de-11e5-8aee-918ac0a06113  Duration: 105.95 ms Billed Duration: 200 ms     Memory Size: 1536 MB    Max Memory Used: 14 MB

Answer 1

I was also following your first link and managed to import numpy and pandas in a Lambda function this way (on Windows):

1. Started a (free-tier) t2.micro EC2 instance with 64-bit Amazon Linux AMI 2015.09.1 and used Putty to SSH in.

2. Tried the same commands you used and the one recommended by the Amazon article:

   sudo yum -y update
   sudo yum -y upgrade
   sudo yum -y groupinstall "Development Tools"
   sudo yum -y install blas --enablerepo=epel
   sudo yum -y install lapack --enablerepo=epel
   sudo yum -y install Cython --enablerepo=epel
   sudo yum install python27-devel python27-pip gcc
   

3. Created the virtual environment:

   virtualenv ~/env
   source ~/env/bin/activate
   

4. Installed the packages:

   sudo ~/env/bin/pip2.7 install numpy
   sudo ~/env/bin/pip2.7 install pandas
   

5. Then, using WinSCP, I logged in and downloaded everything (except _markerlib, pip*, pkg_resources, setuptools* and easyinstall*) from /home/ec2-user/env/lib/python2.7/dist-packages, and everything from /home/ec2-user/env/lib64/python2.7/site-packages from the EC2 instance.

6. I put all these folders and files into one zip, along with the .py file containing the Lambda function. illustration of all files copied

7. Because this .zip is larger than 10 MB, I created an S3 bucket to store the file. I copied the link of the file from there and pasted at "Upload a .ZIP from Amazon S3" at the Lambda function.

8. The EC2 instance can be shut down, it's not needed any more.

With this, I could import numpy and pandas. I'm not familiar with moviepy, but scipy might already be tricky as Lambda has a limit for unzipped deployment package size at 262 144 000 bytes. I'm afraid numpy and scipy together are already over that.

Answer 2

With the help of all posts in this thread here is a solution for the records:

To get this to work you'll need to:

1. start a EC2 instance with at least 2GO RAM (to be able to compile NumPy & SciPy)

2. Install the needed dependencies

   sudo yum -y update
   sudo yum -y upgrade
   sudo yum -y groupinstall "Development Tools"
   sudo yum -y install blas --enablerepo=epel
   sudo yum -y install lapack --enablerepo=epel
   sudo yum -y install Cython --enablerepo=epel
   sudo yum install python27-devel python27-pip gcc
   virtualenv ~/env
   source ~/env/bin/activate
   pip install scipy
   pip install numpy
   pip install moviepy
   

3. Copy to your locale machine all the content of the directories (except _markerlib, pip*, pkg_resources, setuptools* and easyinstall*) in a stack folder:

   • home/ec2-user/env/lib/python2.7/dist-packages
   • home/ec2-user/env/lib64/python2.7/dist-packages

4. get all required shared libraries from you EC2instance:

   • libatlas.so.3
   • libf77blas.so.3
   • liblapack.so.3
   • libptf77blas.so.3
   • libcblas.so.3
   • libgfortran.so.3
   • libptcblas.so.3
   • libquadmath.so.0

5. Put them in a lib subfolder of the stack folder

6. imageio is a dependency of moviepy, you'll need to download some binary version of its dependencies: libfreeimage and of ffmpeg; they can be found here. Put them at the root of your stack folder and rename libfreeimage-3.16.0-linux64.soto libfreeimage.so

7. You should now have a stack folder containing:

   • all python dependencies at root
   • all shared libraries in a lib subfolder
   • ffmpeg binary at root
   • libfreeimage.so at root

8. Zip this folder: zip -r9 stack.zip . -x ".*" -x "*/.*"

9. Use the following lambda_function.py as an entry point for your lambda

   from __future__ import print_function
   
   import os
   import subprocess
   
   SCRIPT_DIR = os.path.dirname(os.path.abspath(__file__))
   LIB_DIR = os.path.join(SCRIPT_DIR, 'lib')
   FFMPEG_BINARY = os.path.join(SCRIPT_DIR, 'ffmpeg')
   
   
   def lambda_handler(event, context):
       command = 'LD_LIBRARY_PATH={} IMAGEIO_FFMPEG_EXE={} python movie_maker.py'.format(
           LIB_DIR,
           FFMPEG_BINARY,
       )
       try:
           output = subprocess.check_output(command, shell=True)
           print(output)
       except subprocess.CalledProcessError as e:
           print(e.output)
   

10. write a movie_maker.pyscript that depends on moviepy, numpy, ...

11. add those to script to your stack.zip file zip -r9 lambda.zip *.py

12. upload the zip to S3 and use it as a source for your lambda

You can also download the stack.zip here.

Answer 3

The posts here help me to find a way to statically compile NumPy with libraries files that can be included in the AWS Lambda Deployment package. This solution does not depend on LD_LIBRARY_PATH value as in @rouk1 solution.

The compiled NumPy library can be downloaded from https://github.com/vitolimandibhrata/aws-lambda-numpy

Here are the steps to custom compile NumPy

Instructions on compiling this package from scratch

Prepare a fresh AWS EC instance with AWS Linux.

Install compiler dependencies

sudo yum -y install python-devel
sudo yum -y install gcc-c++
sudo yum -y install gcc-gfortran
sudo yum -y install libgfortran

Install NumPy dependencies

sudo yum -y install blas
sudo yum -y install lapack
sudo yum -y install atlas-sse3-devel

Create /var/task/lib to contain the runtime libraries

mkdir -p /var/task/lib

/var/task is the root directory where your code will reside in AWS Lambda thus we need to statically link the required library files in a well known folder which in this case /var/task/lib

Copy the following library files to the /var/task/lib

cp /usr/lib64/atlas-sse3/liblapack.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libptf77blas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libf77blas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libptcblas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libcblas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libatlas.so.3 /var/task/lib/.
cp /usr/lib64/atlas-sse3/libptf77blas.so.3 /var/task/lib/.
cp /usr/lib64/libgfortran.so.3 /var/task/lib/.
cp /usr/lib64/libquadmath.so.0 /var/task/lib/.

Get the latest numpy source code from http://sourceforge.net/projects/numpy/files/NumPy/

Go to the numpy source code folder e.g numpy-1.10.4 Create a site.cfg file with the following entries

[atlas]
libraries=lapack,f77blas,cblas,atlas
search_static_first=true
runtime_library_dirs = /var/task/lib
extra_link_args = -lgfortran -lquadmath

-lgfortran -lquadmath flags are required to statically link gfortran and quadmath libraries with files defined in runtime_library_dirs

Build NumPy

python setup.py build

Install NumPy

python setup.py install

Check whether the libraries are linked to the files in /var/task/lib

ldd $PYTHON_HOME/lib64/python2.7/site-packages/numpy/linalg/lapack_lite.so

You should see

linux-vdso.so.1 =>  (0x00007ffe0dd2d000)
liblapack.so.3 => /var/task/lib/liblapack.so.3 (0x00007ffad6be5000)
libptf77blas.so.3 => /var/task/lib/libptf77blas.so.3 (0x00007ffad69c7000)
libptcblas.so.3 => /var/task/lib/libptcblas.so.3 (0x00007ffad67a7000)
libatlas.so.3 => /var/task/lib/libatlas.so.3 (0x00007ffad6174000)
libf77blas.so.3 => /var/task/lib/libf77blas.so.3 (0x00007ffad5f56000)
libcblas.so.3 => /var/task/lib/libcblas.so.3 (0x00007ffad5d36000)
libpython2.7.so.1.0 => /usr/lib64/libpython2.7.so.1.0 (0x00007ffad596d000)
libgfortran.so.3 => /var/task/lib/libgfortran.so.3 (0x00007ffad5654000)
libm.so.6 => /lib64/libm.so.6 (0x00007ffad5352000)
libquadmath.so.0 => /var/task/lib/libquadmath.so.0 (0x00007ffad5117000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007ffad4f00000)
libc.so.6 => /lib64/libc.so.6 (0x00007ffad4b3e000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007ffad4922000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007ffad471d000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007ffad451a000)
/lib64/ld-linux-x86-64.so.2 (0x000055cfc3ab8000)

Answer 4

Another, very simple method that's possible these days is to build using the awesome docker containers that LambCI made to mimic Lambda: https://github.com/lambci/docker-lambda

The lambci/lambda:build container resembles AWS Lambda with the addition of a mostly-complete build environment. To start a shell session in it:

docker run -v "$PWD":/var/task -it lambci/lambda:build bash

Inside the session:

export share=/var/task
easy_install pip
pip install -t $share numpy

Or, with virtualenv:

export share=/var/task
export PS1="[\u@\h:\w]\$ " # required by virtualenv
easy_install pip
pip install virtualenv
# ... make the venv, install numpy, and copy it to $share

Later on you can use the main lambci/lambda container to test your build.

Answer 5

As of 2017, NumPy and SciPy have wheels that work on Lambda (the packages include precompiled libgfortran and libopenblas). As far as I know, MoviePy is a pure Python module, so basically you could do:

pip2 install -t lambda moviepy scipy

Then copy your handler into the lambda directory and zip it. Except, that you'll most likely exceed the 50/250 MB size limits. There are a couple of things that can help:

• remove .pycs, docs, tests and other unnecessary parts;
• leave a single copy of common libraries of NumPy and SciPy;
• strip libraries of inessential pieces, such as debugging symbols;
• compress the archive using higher settings.

Here's an example script that automates the above points.

Answer 6

As of 2018, Steps to install external modules in Python3 on AWS EC2:

1. Launch EC2 on Amazon Linux AMI 201709.

2. ssh with putty using private and public key and become super user.

3. Install Python 3 and create virtual env, then make it default

   yum install python36 python36-virtualenv python36-pip
   
   virtualenv -p python3.6 /tmp/my_python_lib
   
   source /tmp/my_python_lib/bin/activate
   
   which python --to check which version s installed
   
   pip3 install  numpy
   

4. Copy files under site packages and dist packages into your local machhine using winscp.

   To find actual location use grep commands ---

     grep -r dist-packages *. 
   

These packages could be inside both lib and lib64.

5. Site and dist packages will be under location:

   /tmp/my_python_lib/lib64/python3.6,
   /tmp/my_python_lib/lib/python3.6
   

6. Zip these packages along with your script file and upload to S3 which can be accessed in lambda.Instead of zipping the root folder you have to select all files and zip it or send to compressed folder.

Additional tips:

1. If you want to install all packages under one directory, you can use command:

    pip install --upgrade --target=/tmp/my_python_lib/lib/python3.6/dist-packages pandas
   

Answer 7

As of August 2018, probably the easiest way is to start a new AWS Cloud9 environment. Then create a Lambda function inside the environment. Next run this into the Cloud9 command line:

    cd YourApplicationName
    /venv/bin/pip install scipy -t .
    /venv/bin/pip install numpy -t .
    /venv/bin/pip install moviepy -t .

Now I am able to import the modules in the lambda_handler function.

Answer 8

Nov 2018. Hi friends, this post is extremely helpful for me. However, the answers so far are not very automated. I wrote a Python script and tutorial here https://gist.github.com/steinwaywhw/6a6a25d594cc07146c60af943f74c16f to automate the creation of compiled Python packages using pip and virtualenv on EC2. Everything is Python (Boto3), no bash script, no Web console, no awscli.

There are one other change besides automation, which I think is an improvement. I downloaded the whole Python virtual environment from EC2 preserving its folder structures, instead of merging lib and lib64 packages all together. I never understand the intended meaning of merging those two folders. What if some packages override other packages, right? Plus, faking an official virtual environment is non-the-less a safer way to go than rolling your own.

For the downloaded virtual environment to work, the source code of the Lambda function adds some boilerplate code to update Python search path using sys.path. The intended sys.path of a Python virtual environment can be found by

• On your own machine, create a virtual environment and activate it.
• Run a Python script in this virtual environment and do print(sys.path) after import sys. You can start from there and modify as you see fit.

A snippet of the boilerplate code to add for a Lambda function in order to load numpy and other packages from my packaged virtual environment is pasted below. In my case, I loaded pandas_datareader which relies on numpy.

import os
import sys 

# https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html
workdir = os.getenv('LAMBDA_TASK_ROOT')
version = f'{sys.version_info[0]}.{sys.version_info[1]}'
additionals = [f'{workdir}/venv/lib64/python{version}/site-packages',
               f'{workdir}/venv/lib64/python{version}/lib-dynload',
               f'{workdir}/venv/lib64/python{version}/dist-packages',
               f'{workdir}/venv/lib/python{version}/dist-packages',
               f'{workdir}/venv/lib/python{version}/site-packages']
sys.path = additionals + sys.path

import pandas_datareader as pdr

Answer 9

I can confirm that the steps posted by @attila-tanyi work correctly under Amazon Linux. I would only add that there is no need to use an EC2, as there is an Amazon Linux docker container available from the default repository.

docker pull amazonlinux && docker run -it amazonlinux
# Follow @attila-tanyi steps
# Note - sudo is not necessary here

I use the Dockerfile embedded in my application to build and deploy to Lambda.

Answer 10

I like @Vito Limandibhrata's answer but I think it's not enough to build numpy with runtime_library_dirs in numpy==1.11.1. If anybody think site-cfg is ignored, do the following:

cp /usr/lib64/atlas-sse3/*.a /var/task/lib/

*.a files under atlas-sse3 are needed to build numpy. Also, you might need to run the following:

python setup.py config

to check numpy configuration. If it requires something more, you will see the following message:

atlas_threads_info:
Setting PTATLAS=ATLAS   libraries ptf77blas,ptcblas,atlas not found in /root/Envs/skl/lib
    libraries lapack_atlas not found in /root/Envs/skl/lib
    libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib64   
    libraries lapack_atlas not found in /usr/local/lib64
    libraries ptf77blas,ptcblas,atlas not found in /usr/local/lib         
    libraries lapack_atlas not found in /usr/local/lib
    libraries lapack_atlas not found in /usr/lib64/atlas-sse3
<class 'numpy.distutils.system_info.atlas_threads_info'>
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
Setting PTATLAS=ATLAS
    libraries lapack not found in ['/var/task/lib']
Runtime library lapack was not found. Ignoring
    libraries f77blas not found in ['/var/task/lib']
Runtime library f77blas was not found. Ignoring
    libraries cblas not found in ['/var/task/lib']
Runtime library cblas was not found. Ignoring
    libraries atlas not found in ['/var/task/lib']
Runtime library atlas was not found. Ignoring
    FOUND:
        extra_link_args = ['-lgfortran -lquadmath']
        define_macros = [('NO_ATLAS_INFO', -1)]
        language = f77
        libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas', 'lapack', 'f77blas', 'cblas', 'atlas']
        library_dirs = ['/usr/lib64/atlas-sse3']
        include_dirs = ['/usr/include']

then site-cfg is going to be ignored.

Tip: If pip is used to build numpy with runtime_library_dirs, you would better create ~/.numpy-site.cfg and add the following:

[atlas]
libraries = lapack,f77blas,cblas,atlas
search_static_first = true
runtime_library_dirs = /var/task/lib
extra_link_args = -lgfortran -lquadmath

then numpy recognizes .numpy-site.cfg file. It's quite simple and easy way.

Answer 11

You can create zip file for lambda with scipy, numpy, moviepy and pandas on any OS.

https://pypi.org/project/scipy/#files

pypi has wheel files for different OS and you can download manylinux whl file and unzip. After that remove dist-info and pyc files and zip all. The final zip file can be uploaded to S3 then and converted into lambda layer.

This tutorial is helpful to understand it deeply.