0

I got an error while I was trying to import pandas. I created a layer by uploading my package pandas zipped (from windows). I think it's because I zipped my package on windows I don't have the correct structure (same structure as I could get on Linux if zip a file). I read a lot of tutorials, but there are all based on mac or Linux, someone has an idea, to fix this issue?

Lin Du
  • 88,126
  • 95
  • 281
  • 483
Corentin Moreau
  • 111
  • 1
  • 1
  • 12
  • 2
    This needs more information, what error did you get? what did you see? What did you try? Did you find a tutorial about it and followed steps but got stuck somewhere? Yes retry what you did and note down all the steps. and note down the error you got. Google that error and see what you can find. post these findings. – Jasper Lankhorst Nov 09 '19 at 15:06
  • 1
    Build your lambda deployment package in a linux docker container to circumvent the issue that if you package pandas you also package incompatible binaries for a lambda that runs on linux – Drey Nov 09 '19 at 15:19
  • This is a duplicate. Check out my answer here https://stackoverflow.com/a/57837559/112233 – jmp Nov 10 '19 at 00:56
  • Well, first of all how are you deploying your infrastructures? Are you using cloudformation, SAM , Terraform, etc...? I suggest you use AWS or SAM cli to package and deploy your code. Check this script: https://gist.github.com/MatteoGioioso/6fb37ff5a3ad71a39ced625609c554fc – Matteo Nov 10 '19 at 02:34
  • I'd be cautious importing a package as large as pandas into Lambda. If you genuinely do need pandas then research how to minimize its size. If you're just using it for some minor convenience function then consider writing that for yourself. – jarmod Nov 10 '19 at 16:49

1 Answers1

0

I have done work like this. Launch an EC2 with AWS Linux2 then using python3 you create a venv and pip pandas.

python3 -m venv /path/to/new/virtual/environment https://docs.python.org/3/library/venv.html

If pandas and it dependancies are native, it will be created. Which is probably why the zip created on the windows box failed. My fail was I created the venv with dependancies on OSX, also not usable on AWS Linux :-D

You might need to install the development bundle.

yum groupinstall "Development tools"

Follow the instructions here to create the zip file: https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

Take the zip file and save it to S3.

As you update the main lambda function, pull the zip from S3 and update your zip and push to lambda

If new packages are needed, recreate the base zip that is stored on S3.

I wrote code to do all this for work. Hence I cannot just provide the python I wrote that deals with all this. Since, work.

If you you want to build and test this in docker, use this image: https://github.com/lambci/docker-lambda

mpechner
  • 139
  • 7