9

I am a newbie on the AWS & Python and trying to implement a simple ML recommendation system using AWS Lambda function for self-learning. I am stuck on the packaging the combination of sklearn, numpy and pandas. If combined any two lib means (Pandas and Numpy) or (Numpy and Skype) is working fine and deploy perfectly. Because I am using ML system then i need sklearn (scipy and pandas and numpy) which cannot work and getting this error on aws lambda test. What I have done so far : my deployment package from within a python3.6 virtualenv, rather than directly from the host machine. (have python3.6, virtualenv and awscli already installed/configured, and that your lambda function code is in the ~/lambda_code directory):

  1. cd ~ (We'll build the virtualenv in the home directory)

  2. virtualenv venv --python=python3.6 (Create the virtual environment)

  3. source venv/bin/activate (Activate the virtual environment)

  4. pip install sklearn, pandas, numpy

  5. cp -r ~/venv/lib/python3.6/site-packages/* ~/lambda_code (Copy all installed packages into root level of lambda_code directory. This will include a few unnecessary files, but you can remove those yourself if needed)

  6. cd ~/lambda_code

  7. zip -r9 ~/package.zip . (Zip up the lambda package)

  8. aws lambda update-function-code --function-name my_lambda_function --zip-file fileb://~/package.zip (Upload to AWS)

after that getting this error:

**"errorMessage": "Unable to import module 'index'"**

and

START RequestId: 0e9be841-2816-11e8-a8ab-636c0eb502bf Version: $LATEST
Unable to import module 'index': **Missing required dependencies ['numpy']**

END RequestId: 0e9be841-2816-11e8-a8ab-636c0eb502bf
REPORT RequestId: 0e9be841-2816-11e8-a8ab-636c0eb502bf  Duration: 0.90 ms   Billed Duration: 100 ms     Memory Size: 128 MB Max Memory Used: 33 MB

I have tried this on EC2 instance as well but did not a success.I did the google and read multiple blogs and solution but not worked. Please help me out on this.

chthonicdaemon
  • 19,180
  • 2
  • 52
  • 66
Vinay Mishra
  • 386
  • 2
  • 15
  • Did you check [this](https://stackoverflow.com/questions/34749806/using-moviepy-scipy-and-numpy-in-amazon-lambda) and [this](https://stackoverflow.com/questions/43877692/pandas-in-aws-lambda-gives-numpy-error) – Tarun Lalwani Mar 26 '18 at 09:13
  • yes, already tried, but package needed SKLEARN,Numpy and Pandas with no conflict with version. – Vinay Mishra Mar 28 '18 at 05:34

5 Answers5

1

u are using python 3.6 . so pip3 install numpy should be used, make a try .

Go Go Gadget 2
  • 148
  • 1
  • 2
  • 10
0

You need to make sure all the dependent libraries AND the Python file containing your function are all in one zip file in order for it to detect the correct dependencies.

So essentially, you will need to have Numpy, Panda and your own files all in one zip file before you upload it. Also make sure that your code is referring to the local files (in the same unzipped directory) as dependencies. If you have done that already, the issue is probably how your included libraries gets referenced. Make sure you are able to use the included libraries as a dependency by getting the correct relative path on AWS once it's deployed to Lambda.

Ying Li
  • 2,500
  • 2
  • 13
  • 37
  • But this is isolated to Numpy or other similarly referenced libraries as well? If Numpy and your other libraries and resources are in the same location but others can be referenced, then that's very strange. – Ying Li Mar 28 '18 at 16:10
  • Yes, this is very weird behaviour, I think there is a compatibility issue of Numpy, pandas and sklearn lib. because both pandas and sklearn lib use numpy. – Vinay Mishra Apr 03 '18 at 06:23
0

So like Wai kin chung said, you need to use pip3 to install the libraries.

so to figure out which python version is default you can type: 

which python

or

python -v

So in order to install with python3 you need to type:

python3 -m pip install sklearn, pandas, numpy --user

Once that is done, you can make sure that the packages are installed with:

python3 -m pip freeze

This will show all the python libraries installed with your python model. Once you have the libraries you would want to continue with you regular steps. Of course you would first want to delete everything that you have placed in ~/venv/lib/python3.6/site-packages/*.

cd ~/lambda_code
zip -r9 ~/package.zip
Haris Nadeem
  • 1,322
  • 11
  • 24
0

If you're running this on Windows (like I was), you'll run into an issue with the libraries being compiled on an incompatible OS.

You can use an Amazon Linux EC2 instance, or a Cloud9 development instance to build your virtualenv as detailed above.

Or, you could just download the pre-compiled wheel files as discussed on this post: https://aws.amazon.com/premiumsupport/knowledge-center/lambda-python-package-compatible/

Essentially, you need to go to the project page on https://pypi.org and download the files named like the following:

  • For Python 2.7: module-name-version-cp27-cp27mu-manylinux1_x86_64.whl
  • For Python 3.6: module-name-version-cp36-cp36m-manylinux1_x86_64.whl

Then unzip the .whl files to your project directory and re-zip the contents together with your lambda code.

ashtonium
  • 2,081
  • 1
  • 17
  • 20
0

Was having a similar problem on Ubuntu 18.04. Solved the issue by using python3.7 and pip3.7.

Its important to use pip3.7 when installing the packages, like pip3.7 install numpy or pip3.7 install numpy --user

To install python3.7 and pip3.7 on Ubuntu you can use deadsnakes/ppa

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.7

curl https://bootstrap.pypa.io/get-pip.py -o /tmp/get-pip.py
python3.7 /tmp/get-pip.py

This solution should also work on Ubuntu 16.04.

Levon
  • 10,408
  • 4
  • 47
  • 42