6

I'm trying to import numpy library on a lambda AWS these are the steps:

layer=numpy
mkdir -p $layer/python/lib/python3.7/site-packages/
cd $layer/python/lib/python3.7/site-packages/
pip install -t . numpy
cd ../../../../
zip -r $layer.zip .

Here the error from lambda:

[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function':

IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!

Importing the numpy c-extensions failed. - Try uninstalling and reinstalling numpy. - If you have already done that, then: 1. Check that you expected to use Python3.7 from "/var/lang/bin/python3.7", and that you have no directories in your PATH or PYTHONPATH that can interfere with the Python and numpy version "1.17.3" you're trying to use. 2. If (1) looks fine, you can open a new issue at https://github.com/numpy/numpy/issues. Please include details on: - how you installed Python - how you installed numpy - your operating system - whether or not you have multiple versions of Python installed - if you built from source, your compiler versions and ideally a build log

  • If you're working with a numpy git repository, try git clean -xdf (removes all files not under version control) and rebuild numpy.

Note: this error has many possible causes, so please don't comment on an existing issue about this - open a new one instead.

Original error was: No module named 'numpy.core._multiarray_umath'

Here is the structure of my path:

.
└── python
    └── lib
        └── python3.7
            └── site-packages

Thanks

SOLUTION:

docker pull python:3.7-slim-buster
docker run -it python:3.7-slim-buster bash
cd home

layer=boto3_cv2_pil
mkdir -p $layer/python/lib/python3.7/site-packages/
cd $layer/python/lib/python3.7/site-packages/
pip install -t . boto3 opencv-python Pillow numpy==1.17.3 pytz
cd ../../../../
apt-get update
apt-get install zip -y
zip -r $layer.zip .

docker cp $CONTAINER_ID:/home/boto3_cv2_pil.zip .
ambigus9
  • 1,417
  • 3
  • 19
  • 37

2 Answers2

4

For building custom layers, I can recommend using the following docker tool:

Offical AWS blog explains how to use it:

How do I create a Lambda layer using a simulated Lambda environment with Docker?

The following SO answer is also helpful from my experience:

Marcin
  • 215,873
  • 14
  • 235
  • 294
  • @ambigus9 Let me know how it goes. I successful used this tool to build layers with numpy, pillow, scipy. It may be difficult to start with, as have to setup docker, etc, but in the long term its worth the effort. – Marcin May 20 '20 at 00:20
  • 1
    Your solution inspired me. Thanks! – ambigus9 May 20 '20 at 04:47
  • @ambigus9 No problem. Glad I could help. – Marcin May 20 '20 at 04:57
1

You don't need to build your own NumPy layer. It's already provided by AWS. Check out this blog post.

jellycsc
  • 10,904
  • 2
  • 15
  • 32
  • Sure, However, I need to compile more libraries: `pip install -t . boto3 opencv-python Pillow numpy==1.17.3 pytz` and i'm getting that error. Any idea how to fix it? – ambigus9 May 20 '20 at 00:05
  • @ambigus9 Could you post the file or directory structure of the deployment package? For example, the output of the `tree` command. – jellycsc May 20 '20 at 00:09