aws_cdk python error: Unzipped size must be smaller than 262144000 bytes

Question

I use CDK to deploy a lambda function that uses several python modules. But I got the following error at the deployment.

Unzipped size must be smaller than 262144000 bytes (Service: AWSLambdaInte
rnal; Status Code: 400; Error Code: InvalidParameterValueException;

I have searched following other questions, related to this issue.

question1 question2

But they focus on serverless.yaml and don't solve my problem. Is there any way around for this problem?

Here is my app.py for CDK.

from aws_cdk import (
    aws_events as events,
    aws_lambda as lam,
    core,
)

class MyStack(core.Stack):
    def __init__(self, app: core.App, id: str) -> None:
        super().__init__(app, id)

        layer = lam.LayerVersion(
            self, "MyLayer",
            code=lam.AssetCode.from_asset('./lib'),
        );

        makeQFn = lam.Function(
            self, "Singleton", 
            function_name='makeQ',
            code=lam.AssetCode.from_asset('./code'),
            handler="makeQ.main",
            timeout=core.Duration.seconds(300),
            layers=[layer],
            runtime=lam.Runtime.PYTHON_3_7,
        )


app = core.App()
MyStack(app, "MS")
app.synth()

In ./lib directory, I put python modules like,

python -m pip install numpy -t lib/python

Answer 1

Edit: Better method!

---

Original:

There is now an experimental package, aws_lambda_python_alpha which you can use to automatically bundle packages listed in a requirements.txt, but I'm unfortunately still running into the same size issue. I'm thinking now to try layers.

For anyone curious, here's a sample of bundling using aws_lambda_python_alpha:

from aws_cdk import aws_lambda_python_alpha as _lambda_python

self.prediction_lambda = _lambda_python.PythonFunction(
    scope=self,
    id="PredictionLambda",
    # entry points to the directory
    entry="lambda_funcs/APILambda",
    # index is the file name
    index="API_lambda.py",
    # handler is the function entry point name in the lambda.py file
    handler="handler",
    runtime=_lambda.Runtime.PYTHON_3_9,
    # name of function on AWS
    function_name="ExampleAPILambda",
)

Answer 2

I would suggest checking out the aws-cdk-lambda-asset project, which will help bundle internal project dependencies stored in a requirements.txt file. How this works is, it installs dependencies specified in the requirements file to a local folder, then bundles it up in a zip file which is then used for CDK deployment.

For non-Linux environments like Windows/Mac, it will install the dependencies in a Docker image, so first ensure that you have Docker installed and running on your system.

Note that the above code seems to use poetry, which is a dependency management tool. I have no idea what poetry does or why it is used in lieu of a setup.py file. Therefore, I've created a slight modification in a gist here, in case this is of interest; this will install all local dependencies using a regular pip install command instead of the poetry tool, which I'm not too familiar with.

Answer 3

Thanks a lot.

In my case, the issue is solved just by removing all __pycache__ in the local modules before the deployment.

I hope the situation will be improved and we only have to upload requirements.txt instead of preparing all the modules locally.

Answer 4

Nope, there is no way around that limit in single setup.

What you have to do instead is install your dependencies into multiple zips that become multiple layers.

Basically, install several dependencies to a python folder. Then zip that folder into something like intergration_layer. Clear the python folder and install the next set and name it something else. Like data_manipilation.

Then you have two layers in cdk (using aws_lambda.LayerVersion) and add those layers to each lambda. You'll have to break the layers up to be small enough.

You can use a makefile to generate the layers automatically and then tie the makefile, cdk deploy, and some clean up inside a bash script yo tie them all together.

Note. You are still limited on space with layers and limited to 5 layers. If your dependencies outgrow that then look into Elastic File System. You can install dependencies there and tie that single EFS to any lambda to reference them through PYTHONPATH manipulation built into the EFS lambda connection

Basic makefile idea (but not exact for CDK, but still close) (and if you never have, a good tutorial on how to use a makefile)

CDK documentation for Layers (use from AssetCode for the location)

AWS dev blog on EFS for Lambdas