8

I'm looking for a work around to use numpy in AWS lambda. I am not using EC2 just lambda for this so if anyone has a suggestion that'd be appreciated. Currently getting the error:

cannot import name 'multiarray'

Using grunt lambda to create the zip file and upload the function code. All the modules that I use are installed into a folder called python_modules inside the root of the lambda function which includes numpy using pip install and a requirements.txt file.

Udo Held
  • 12,314
  • 11
  • 67
  • 93
Sebastian Łącki
  • 167
  • 1
  • 2
  • 14
  • You should probably have your modules installed or symlinked directly to the root of your Lambda function package. And make sure that all the dependencies also go there. – Nikolay Grischenko Sep 13 '17 at 04:22
  • I know what you're getting at but all the modules work fine as the env variable points to them. – Sebastian Łącki Sep 14 '17 at 22:26
  • Did you solve this issue by any chance? I have the exact issue on Python 3.6? – JavaCake Nov 11 '17 at 10:28
  • I haven't gotten back to it, but i did find this link althought i haven't had a chance to look back into it! Hopefully you get something out of this and if you do let me know! https://github.com/vitolimandibhrata/aws-lambda-numpy . From my understanding it takes complete versions of numpy packages and you can drop them into your project. – Sebastian Łącki Nov 11 '17 at 16:10
  • As of 2018 it's best and easiest to use layers: https://aws.amazon.com/blogs/aws/new-for-aws-lambda-use-any-programming-language-and-share-common-components/ AWS even have premade one for numpy – DaveRGP May 01 '19 at 13:58

7 Answers7

11

An easy way to make your lambda function support the numpy library for python 3.7:

  1. Go to your lambda function page
  2. Find the Layers section at the bottom of the page.
  3. Click on Add a layer.
  4. Choose AWS layers as layer source.
  5. Select AWSLambda-Python37-Scipy1x as AWS layers.
  6. Select 37 for version.
  7. And finally click on Add.

Now your lambda function is ready to support numpy.

user_5
  • 498
  • 1
  • 5
  • 22
7

Updated to include the solution here, rather than a link:

After much effort, I found that I had to create my deployment package from within a python3.6 virtualenv, rather than directly from the host machine. I did the following within a Ubuntu 16.04 docker image. This assumes that you have python3.6, virtualenv and awscli already installed/configured, and that your lambda function code is in the ~/lambda_code directory:

1) cd ~ (We'll build the virtualenv in the home directory)

2) virtualenv venv --python=python3.6 (Create the virtual environment)

3) source venv/bin/activate (Activate the virtual environment)

4) pip install numpy

5) cp -r ~/venv/lib/python3.6/site-packages/* ~/lambda_code (Copy all installed packages into root level of lambda_code directory. This will include a few unnecessary files, but you can remove those yourself if needed)

6) cd ~/lambda_code

7) zip -r9 ~/package.zip . (Zip up the lambda package)

8) aws lambda update-function-code --function-name my_lambda_function --zip-file fileb://~/package.zip (Upload to AWS)

Your lambda function should now be able to import numpy with no problems.

If you want a more out-of-the-box solution, you could consider using serverless to deploy your lambda function. Before I found the above solution, I followed the guide here and was able to run numpy successfully in a python3.6 lambda function.

cailan s
  • 121
  • 1
  • 4
  • you can save yourself some time by using Docker and/or Serverless framework https://stackoverflow.com/questions/43859497/aws-lambda-and-python-numpy-module/50270737#50270737 – GWed May 10 '18 at 10:30
1

As of 2018 it's best to just use the inbuilt layers functionality.

AWS have actually released a pre-made one with numpy in it: https://aws.amazon.com/blogs/aws/new-for-aws-lambda-use-any-programming-language-and-share-common-components/

DaveRGP
  • 1,430
  • 15
  • 34
0

I was unable to find a good solution using serverless plugins, but I did find a good way with layers. See Serverless - Numpy - Unable to find good bind path format

launchpadmcquack
  • 1,021
  • 1
  • 12
  • 19
0

Add numpy layer in this way:

  • Go on your lambda function

  • select add a new layer

  • add it using this arn: arn:aws:lambda:eu-central-1:770693421928:layer:Klayers-p39-numpy:7

(change your zone if you are not in eu-central-1)

Let me know if it will work

Saro
  • 1
  • 1
-1

I would add this answer as well: https://stackoverflow.com/a/52508839/1073691

Using pipenv includes all of the needed .so files as well.

alexpotato
  • 1,168
  • 2
  • 9
  • 28
-1

1.) Do a Pip install of numpy to a folder on your local machine.

2.) once complete, zip the entire folder and create a zip file.

3.) Go to AWS lambda console, create a layer and upload zip file created in step 2 there and save the layer.

4.) After you create your lambda function, click add layer and add the layer you created. That's it, import numpy will start working.

appleios
  • 9
  • 2