4

I need to use AWS Lambda triggered through API gateway. I have python script which loads a machine learning model from S3bucket and gets input from api call and predicts the result. I can successfully trigger the lambda function written inline in python. But I want to use machine learning packages to predict in lambda function. So I came to know that I need to upload the code with the packages installed in virtual environment and I did.But the lambda when triggered gives the error 'Unable to import model lambda_function'. I have lambda_function.py with method 'handler'. Please let me know if Iam doing it right(creating virtual env and installing packages and uploading it) and why is this error. Also, let me know the solutions for Windows and AWS console. I have seen many answers with Linux commands and using aws cli.

zip folder

lambda_function

lamnda function settings

lambda function settings

Update:

This is driving me crazy!. I have tried all the methods found in the answers and none works for me. And it gives the same error : 'Unable to import module : lambda_function' So Iam not able to understand where the error is. Please help me if you have any suggestion. Before you say function names: I have correct names: lambda_function.lambda_handler. I zipped the contents and not directory. Please see my lambda code and lambda settings below lambda json file lambda function code: import boto3 import os import uuid import sklearn import pickle

def lambda_handler(event, context):
s3_client = boto3.client('s3')
s_desc=event['params']['querystring']['token']
X_test1=[]
X_test1.append(s_desc)
#load model
bucket = 'harshini-snow-bucket'
key = 'model.pkl'
download_path = '/tmp/{}{}'.format(uuid.uuid4(), key)
s3_client.download_file(bucket, key, download_path)

f = open(download_path, 'rb')
model = pickle.load(f)
f.close()
#class_predicted = model.predict(X_test1)

return X_test1

Please tell me if there are any other ways.. I will try anything for this to work.

Update 2:

error

code

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
harshini
  • 77
  • 3
  • 7
  • Looks like you're getting an indentation issue. I get these sometimes when I copy-paste code between files. Try unindenting re-indenting the code in pack.py (your error image shows that the issue is in pack.py). There are also often options in your text editor that can help highlight where spaces may have been used instead of tabs, or if you just forgot to indent somewhere. – zyd Jun 11 '18 at 19:52
  • As you can see the code, I couldn't find any indentation error and retried it several times. – harshini Jun 11 '18 at 20:16
  • You didn't post pack.py... as you can see in your error message, it says there is an indentation issue in pack.py around line 12 (look at the second line in the Log output area of your error image). Sometimes I have to just re-do the indentation because it'll look like the indentation is fine but in actuality a line got copied using spaces instead of tabs or vice versa. The 'unindent does not match any outer indentation level' error you're getting tends more often to mean that the indentation is just wrong though, probably from not indenting under a function or control structure. – zyd Jun 11 '18 at 20:50
  • Please see the link named code which shows the pack.py and check the indentations. – harshini Jun 11 '18 at 20:53
  • Ah sorry I thought that was lambda_function.py. The indentations do look fine from here, but like I said earlier sometimes it looks fine when there are hidden characters causing indentation issues. Did you try un-indenting and re-indenting all the lines in lambda_handler? – zyd Jun 11 '18 at 20:59

6 Answers6

3

First install all the required packages inside a folder in local machine. Also include the main lambda_function.py file inside that folder. Now select all the packages and the python file inside the folder and compress them into a .zip file. Here you have to make sure you compress contents of the folder not the folder itself. Then you can upload the zip file to lambda either directly or through s3. Another point to note is if the python file is named "lambda_function.py" or not. By default lambda assumes the main python file to have "lambda_function.py" name. If you have used any other name, you can change the filename from lambda console under Function code section inside Handler...Hope this helps out.

Manoj Acharya
  • 1,331
  • 2
  • 15
  • 27
  • This tip is really useful for machine learning models since the packages that it uses are usually large. And lambda has a limitation of ~50MB of compressed source code file (As per docs) whereas uncompressed can be around 250 MB. Easiest way is to push the compressed file even if greater than 50 MB to s3 and then refer that during creation of lambda function as source. – Parachute Jun 06 '18 at 20:00
  • Hi I have already followed all the above steps but still facing the error. I have compressed the files not folder and my function name is lambda_function in both. Iam stuck with this error for a long time now. Don't know the reason!!! – harshini Jun 07 '18 at 01:31
  • Here is my zip folder with lambda_function.py and all the other folders are created when I ran the virtual environment and installed packages.( See the my zip link in the question) – harshini Jun 07 '18 at 01:34
  • Try installing each packages seperately inside the folder without the concept of virtual env! For eg: To install boto3 library, pip install boto3 -t /path/to/your/folder – Manoj Acharya Jun 07 '18 at 02:20
  • what will my lambda_function.py ? And how I can upload directly to lambda function . kindly guide please – ephemeral Jun 07 '18 at 07:14
3

You have to make a custom deployment package for lambda either using docker or EC2. It will not work if you make a package in local machine as it will not compile the libaries needed.

here is the complete example, how you will make a custom package , this example packages PILLOW image processing library of python along with the lambda code , you can pack all other libraries you need for your model in the same way in the same package along with PIL

link to example

Remember one thing , in example filename is CreateThumbnail.py, you can give it any name , but always configure your handler in this way filename.handler-function , e.g disco.lambda_handler where disco.py is filename and lambda_handler is code handler module for lambda

Mausam Sharma
  • 852
  • 5
  • 10
  • I followed the exact steps but still get the same error : Unable to import module lambda_function. – harshini Jun 08 '18 at 02:41
  • Is your filename **lambda_function.py** ?? And do your code contains a lambda handler ? – Mausam Sharma Jun 08 '18 at 14:03
  • Yes.. See the images for the created zip folder and my lambda_function and handler name is handler in lambda function. – harshini Jun 08 '18 at 15:10
  • Can you add screenshots of your lambda environment settings in the console ? – Mausam Sharma Jun 09 '18 at 01:33
  • Attached. See the question for images. – harshini Jun 09 '18 at 14:50
  • ok , you are still getting **Unable to import module error**, according to your code i think you need just **sklearn** as an extra library , so i have created a package for you https://s3.amazonaws.com/mausamrest/tomato.zip this package contains all libraries you need except boto3 as it is already available in lambda. Also I have not written the code , i have only created a package with all libraries and a **test run file** called **pack.py** which is working absolutely fine in lambda environment – Mausam Sharma Jun 11 '18 at 13:18
  • here is the settings you need to do in your console : **1**. make sure you have **python 3.6** as runtime **2** Handler looks like this https://s3.amazonaws.com/mausamrest/harshini.JPG **here is the result what i got on runtime** https://s3.amazonaws.com/mausamrest/harshini2.JPG unzip the folder, make changes in **pack.py** ,make zip again of only the contents, upload it to s3 bucket , and from there provide link of your zip in lambda – Mausam Sharma Jun 11 '18 at 13:23
  • Thank you so much for taking time to solve this. I used your package and modified the package and good news is I don't get the unable to import error but now I get syntax error in module 'pack'. I have checked so many times and there is no syntax error. – harshini Jun 11 '18 at 19:29
  • import boto3 import os import uuid import pickle import sklearn def lambda_handler(event,context): s3_client = boto3.client('s3') s_desc=event['params']['querystring']['token'] #X_test1=[] #X_test1.append(s_desc) #load model bucket = 'harshini-snow-bucket' key = 'model.pkl' download_path = '/tmp/{}{}'.format(uuid.uuid4(), key) s3_client.download_file(bucket, key, download_path) f = open(download_path, 'rb') model = pickle.load(f) f.close() #class_predicted = model.predict(X_test1) print("mausam") return s_desc – harshini Jun 11 '18 at 19:31
  • **1**. No need to enter code in comments as it is useless , "*2**. Just by visually seeing the code, no one can say whether there are indentation errors or not., **3** I think you must have copied your code in my file and that copying usually creates indentation error, hence from handler function , write your function line by line and then try again – Mausam Sharma Jun 12 '18 at 01:01
  • Finally!! Got it! Firstly thank you for zip file of packages you provided, once I had that error changed. Then with the other error, it was problem with this line: model = pickle.load(f) I saved the model with joblib package and was loading it with pickle. There are sources which says both are almost same but when I changed to loaded_model = joblib.load('/tmp/model.pkl') , it worked perfectly. – harshini Jun 12 '18 at 02:13
  • Please let me know the process you did to zip the packages.. and please let me know how to accept the answer, Iam not able to vote it as I have less reputation. Is there any other to mark it as correct? – harshini Jun 12 '18 at 02:16
  • https://drive.google.com/file/d/1GH3EhQ4hvmzGXQuLIHH6GAk61Jx3iWku/view?usp=sharing by clicking on the grey tick , making it green , you can accept the answer and I followed the procedure which i have mentioned as an example in my answer , this example I gave to make a package of PIL library for which someone else has asked question – Mausam Sharma Jun 12 '18 at 09:13
  • Thank you. Did that. And I followed the same procedure but couldn't get it. – harshini Jun 12 '18 at 14:55
  • Yeah..my mistake was in packages. I removed all the unnecessary packages not needed for my function and it worked. – harshini Jun 15 '18 at 01:47
0

The answers here are technically correct but if you're working on a Windows machine (or anything other than Amazon Linux), you might have a lot of issues putting custom packages in your Python app. Lambda apps run on Amazon Linux so you need to install the packages using that OS or anything as close as possible.

Here's one of my answers which describes how I put together packages for my Lambda app:

https://stackoverflow.com/a/50767639/3023353

phoenix
  • 1,629
  • 20
  • 11
0

I know that i am a bit late for answering this question but I was facing the same problem with pandas library for python.

this was the link which help me to solve this issue HERE

so the answer in a nutshell is

1.go to here

2.search the library you want to use

3.then go to download option

4.download the file with Linux option

5.Unzip the file in your project folder (I use 7-zip for that)

6.now make a new zip of your project and upload on lambda

annish kumar
  • 23
  • 2
0

You need to add a Layer for outside python libraries in AWS Lambda.

https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html

The AWS documentation is very bad with examples and for our team it was not easy. Had to manually install a number of libraries but this worked.

Mike
  • 43
  • 8
0

This drove me crazy too. I am using Windows to create the python packages. The key to solving this is to store your packages in a folder named python. When I named the folder anything else it wouldnt work.

Scott
  • 31
  • 3