Import libraries in lambda layers

Question

I wanted to import jsonschema library in my AWS Lambda in order to perform request validation. Instead of bundling the dependency with my app , I am looking to do this via Lambda Layers. I zipped all the dependencies under venv/lib/python3.6/site-packages/. I uploaded this as a lambda layer and added it to my aws lambda using publish-layer-version and aws lambda update-function-configuration commands respectively. The zip folder is name "lambda-dep.zip" and all the files are under it. However when I try to import jsonschema in my lambda_function , I see the error below -

from jsonschema import validate

{
  "errorMessage": "Unable to import module 'lambda_api': No module named 'jsonschema'",
  "errorType": "Runtime.ImportModuleError"
}

Am I missing any steps are is there a different mechanism to import anything within lambda layers?

score 79 · Accepted Answer · answered Apr 15 '19 at 19:30

79

You want to make sure your .zip follows this folder structure when unzipped

python/lib/python3.6/site-packages/{LibrariesGoHere}.

Upload that zip, make sure the layer is added to the Lambda function and you should be good to go.

This is the structure that has worked for me.

answered Apr 15 '19 at 19:30

guitarhero23

1,065
9
11

Thanks I'll try this! – Punter Vicky Apr 15 '19 at 19:33
The zip's name is lambda_dependencies. So when I unzipped it created this folder structure - python/lib/python3.6/site-packages/{LibrariesGoHere}. However I still see the issue. Am I missing something? – Punter Vicky Apr 15 '19 at 20:07
3

This works , I changed the folder name to python3.7 since I am using python 3.7 in lambda. Thank you! – Punter Vicky Apr 15 '19 at 20:40
2

Sweet! Yea I would have mentioned to make sure you use the correct python version but in your original question it mentioned python 3.6 so figured that's what you were on. Congrats on getting it working. Also thank you, I can finally comment on things now lol. I've been trying to get an accepted answer forever because of the limitations put on my account until i had reputation points. – guitarhero23 Apr 16 '19 at 12:35
Great! I hope your answers more people like me :) is this way of packaging lambda layers documented anywhere ? I was curious if I was missing to lookup any documentation that is available. – Punter Vicky Apr 16 '19 at 13:22
8

I had looked at a few things but at 1:42 of this video is what finally made me understand it https://youtu.be/ebhcs-9FYJA?t=102 – guitarhero23 Apr 16 '19 at 14:44
Thank you very much – Punter Vicky Apr 16 '19 at 14:45
This solved my problem. I could not find this solution anywhere else or in AWS's docs. – user3454396 Nov 18 '19 at 16:11
Glad to hear it @user3454396, I wish they wrote it out as clearly as this, I looked all around and found a video on youtube that briefly showed the proper folder structure for the various languages. Not very clear in AWS documentation – guitarhero23 Nov 18 '19 at 17:03
According to the video they should go in either `/opt/python` or `/opt/python/lib/python3.6/site-packages`. The video also gives the indication that if you're using 3.7 or above you have to use the second one but I tried it in just `/opt/python` with 3.8 and it seemed to work. I feel like for simplicity it would be easier to just put in `/opt/python` but if you're making a layer for multiple versions then maybe you would want to do the second. – Hayden Feb 14 '20 at 22:55
1

Here is docs about directory layout: https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-path – Pawel Oct 14 '20 at 09:59
This also helped me: https://www.youtube.com/watch?v=3BH79Uciw5w. The zip file needs to be NAMED EXACTLY: python.zip. – Makaroni Feb 26 '21 at 14:34
This solution worked for me as compared to other recommended ones. For Windows, 1) Use conda 2) install python 3.7 – Jag Oct 21 '21 at 01:08
I missed 5 minutes deadline to edit and provide comprehensive - This solution worked for me as compared to other recommended ones. For Windows, 1) Use conda 2) Create environment with python 3.7 3) activate 4) pip install pymysql -t . 5) zip it. Please note it is likely you will have change the path to suit suggested python/lib/python3.6/site-packages/{LibrariesGoHere}. – Jag Oct 21 '21 at 01:17
1

Per AWS documentation, requirements have been changed to simply be placed in a /python directory without the rest of the directory structure. https://aws.amazon.com/premiumsupport/knowledge-center/lambda-import-module-error-python/ – Ryan Lutz May 31 '22 at 19:55

score 15 · Answer 2 · answered Apr 15 '19 at 20:17

Here the script that I use to upload a layer:

#!/usr/bin/env bash

LAYER_NAME=$1 # input layer, retrived as arg
ZIP_ARTIFACT=${LAYER_NAME}.zip
LAYER_BUILD_DIR="python"

# note: put the libraries in a folder supported by the runtime, means that should by python

rm -rf ${LAYER_BUILD_DIR} && mkdir -p ${LAYER_BUILD_DIR}

docker run --rm -v `pwd`:/var/task:z lambci/lambda:build-python3.6 python3.6 -m pip --isolated install -t ${LAYER_BUILD_DIR} -r requirements.txt

zip -r ${ZIP_ARTIFACT} .

echo "Publishing layer to AWS..."
aws lambda publish-layer-version --layer-name ${LAYER_NAME} --zip-file fileb://${ZIP_ARTIFACT} --compatible-runtimes python3.6

# clean up
rm -rf ${LAYER_BUILD_DIR}
rm -r ${ZIP_ARTIFACT}

I added the content above to a file called build_layer.sh, then I call it as bash build_layer.sh my_layer. The script requires a requirements.txt in the same folder, and it uses Docker to have the same runtime used for Python3.6 Lambdas. The arg of the script is the layer name.

After uploading a layer to AWS, be sure that the right layer's version is referenced inside your Lambda.

Thank you very much @nicor88 – Punter Vicky Apr 15 '19 at 20:41 — Punter Vicky, Apr 15 '19 at 20:41
Does anyone know if they'll support python3.9? – Fabio Milheiro Mar 21 '23 at 10:21 — Fabio Milheiro, Mar 21 '23 at 10:21

score 12 · Answer 3 · answered May 31 '22 at 21:02

Update from previous answers: Per AWS documentation, requirements have been changed to simply be placed in a /python directory without the rest of the directory structure.

https://aws.amazon.com/premiumsupport/knowledge-center/lambda-import-module-error-python/

Be sure your unzipped directory structure has libraries within a /python directory.

score 4 · Answer 4 · edited Dec 02 '20 at 16:02

4

There is an easier method. Just install the packages into a python folder. Then install the packages using the -t (Target) option. Note the "." in the zip file. this is a wild card.

mkdir lambda_function
cd lambda_function
mkdir python
cd python
pip install yourPackages -t ./
cd ..
zip /tmp/labmda_layer.zip .

The zip file is now your lambda layer.

The step by step instructions includeing video instructions can be found here.

https://geektopia.tech/post.php?blogpost=Create_Lambda_Layer_Python

edited Dec 02 '20 at 16:02

MikeW

5,504
1
34
29

answered Apr 21 '19 at 08:01

Joseph

512
2
5

3

That is exactly the same as the script that I posted. But you also have to consider that pip install needs to run in the right OS. If you build a layer in OSx with pandas, it won't work. – nicor88 Apr 24 '19 at 15:23
I like this answer because it seems fairly independent of the python version being used. (Not sure if it would work to just take guitarhero23s answer above and replace 3.6 with 3.8, for example). I just used this with 3.8 and it works great though. – Maile Cupo Dec 17 '20 at 19:13
Good answer. Adding "-t ." after pip install is extremely important and the layer won't work if you install without adding it. – ire Jul 08 '22 at 09:48

Import libraries in lambda layers

4 Answers4

Linked