2

I am trying to run the cvxpy package in an AWS lambda function. This package isn't in the SDK, so I've read that I'll have to compile the dependencies into a zip, and then upload the zip into the lambda function.

I've done some research and tried out the links below, but when I try to pip install cvxpy I get error messages - I'm on a Windows box, but I know that AWS Lambda runs on Linux.

Appreciate the help!

http://i-systems.github.io/HSE545/machine%20learning%20all/cvxpy_install/CVXPY%2BInstallation%2BGuide%2Bfor%2BWindows.html

https://programwithus.com/learn-to-code/Pip-and-virtualenv-on-Windows/

https://medium.com/@manivannan_data/import-custom-python-packages-on-aws-lambda-function-5fbac36b40f8

https://www.cvxpy.org/install/index.html

user10756193
  • 183
  • 1
  • 1
  • 8

2 Answers2

2

For installing cvxpy on windows it requires c++ build tools (please refer: https://buildmedia.readthedocs.org/media/pdf/cvxpy/latest/cvxpy.pdf)

On Windows:

pip install cvxpy --target python/lib/python3.7/site-packages

On Linux:

  • I created the same directory structure as earlier python/lib/python3.7/site-packages and installed the cvxpy and zipped it as shown below.
  • Later I uploaded the zip file to an S3 bucket and created a new lambda layer.
  • Attaching that lambda layer to my lambda function, I colud able to resolve the import issues failing earlier and run the basic cvxpy program on lambda. 
mkdir -p alley/python/lib/python3.7/site-packages
pip install cvxpy --target alley/python/lib/python3.7/site-packages
cd alley
zip -rqvT cvxpy_layer.zip .

Lambda layer Image:

enter image description here

Lambda function execution:

enter image description here

Rineesh Konaparthi
  • 104
  • 4
  • 1
    under the hood, AWS lambda run Amazon Linux, hence any package compiled on Windows won't work. That's why you need to compile it on a linux. There's a lambci docker container that can help you do this. – keithRozario Nov 04 '19 at 14:49
  • I stuck at step pip install cvxpy with error "Failed building wheel for scs". – YuMei Sep 26 '21 at 17:25
0

You can wrap all your dependencies along with lambda source into a single zipfile and deploy it. Doing this, you will end up having additional repetitive code in multiple lambda functions. Suppose, if more than one of your lambda functions needs the same package cvxpy, you will have to package it twice for both the functions individually.

Instead a better option would be to try Labmda Layers, where you put all your dependencies into a package and deploy a layer into your Lambda. Then attach that layer to your function to fetch its dependencies from there. The layers can even be versioned. :)

Please refer the below links:

Rineesh Konaparthi
  • 104
  • 4
  • Do you have another example you could provide? I'm specifically trying to use cvxpy and I'm unable to install and package it. – user10756193 Oct 31 '19 at 17:34