Nodejs modularized aws-sdk v3 size getting increased

Question

I am trying to reduce size of nodejs lambda bundle which uses aws-sdk. This is the original lambda package.json file:

{
  "name": "lambdanodejs",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "aws-sdk": "^2.784.0",
    "bluebird": "^3.7.2",
    "ioredis": "^4.19.2",
    "redis": "^3.0.2",
    "redis-clustr": "^1.7.0"
  }
}

Overall size is 57MB, 54 belongs to aws-sdk.

To reduce size I tried using specific client services (v3 sdk). Followed : https://github.com/aws/aws-sdk-js-v3/blob/main/clients/client-secrets-manager/package.json

{
  "name": "lambdanodejs",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "@aws-sdk/client-dynamodb": "^3.7.0",
    "@aws-sdk/client-secrets-manager": "^3.7.0",
    "bluebird": "^3.7.2",
    "ioredis": "^4.19.2",
    "redis": "^3.0.2",
    "redis-clustr": "^1.7.0"
  }
}

Now npm install results in even larger size around 190+MBs. Also in node_modules I see lot of directories which were not there when using previous package.json install. This v3 aws-sdk is supposed to be lighter. Am I missing something?

Thanks!

Can you share how you are bundling your artifact? Or do you just zip your source folder including the `node_modules` folder? — s.hesse, Mar 05 '21 at 08:10
@s.hesse I am zipping node_modules along with other files. These mentioned sizes are before zipping, but according to articles/blogs it should have lesser size, at least compared to aws-sdk it should be much lesser. — greendragons, Mar 05 '21 at 08:16
Ok.. Did you clean the `node_modules` folder before migrating to the new SDK? Maybe the old SDK folder is still there. — s.hesse, Mar 05 '21 at 08:18
@s.hesse Yes I've removed package-lock.json and node_modules both. — greendragons, Mar 05 '21 at 08:22
Interesting.. so I just took your example `package.json` (with AWS SDK v3) and ran `npm install` and I have ~22MB of data in `node_modules`. Not sure what else you could do about it... Alternatively, try using something like `esbuild` to generate one artifact -> it'll bundle everything into one file and throw out all unnecessary code. — s.hesse, Mar 05 '21 at 08:45
@s.hesse Yes, even my friend tried on windows machine and getting around same size ~20MB, on my machine however it is showing up 191MB bulky node_modules. — greendragons, Mar 05 '21 at 09:04
The @aws-sdk/client-secrets-manager library had huge size initially and I too faced the same issue, but as of now, after using the latest version the size of the node_modules has decreased considerably. Please upgrade the package version and try, it should be effective. — harishanth raveendren, Aug 08 '23 at 11:33

score 1 · Answer 1 · answered Mar 05 '21 at 09:13

1

Don't push aws-sdk to the lambda function. If you want to use it in the local development environment install it globally.
Lambda will provide all necessary SDK modules(v2 & v3) during runtime, just import them in the code.
Don't bundle the aws-sdk in the zip.
If your zip size is getting increased, use layers in lambda.
Import the node_moudles to lambda layers.
Then upload your code to lambda.

answered Mar 05 '21 at 09:13

Siva Sumanth

607
4
10

You mean I don't need package.json for aws modules, I can directly invoke import from code and those modules are available @ runtime? – greendragons Mar 05 '21 at 10:13
Don't include aws-sdk or any version of aws-sdk in package.json and project node_modules – Siva Sumanth Mar 05 '21 at 10:22
Assign s3 put object role to your lambda, write the code in lambda console for s3 put object and execute it. for testing purpose try this – Siva Sumanth Mar 05 '21 at 10:24
Will try it out without any external package bundling. – greendragons Mar 05 '21 at 10:25
3

It seems like the lambda nodejs environment comes with aws-sdk v2 pre-installed, but not v3: https://stackoverflow.com/questions/66129993/does-aws-lambda-support-aws-sdk-v3-or-not – zirkelc Jul 06 '21 at 15:55
Yes, v2 pre-installed. – Siva Sumanth Jul 06 '21 at 16:09
Installing a library globally in node is a really bad practice. Maybe consider something like yarn plug and play https://yarnpkg.com/features/pnp – Gabriel Furstenheim Dec 22 '21 at 10:48

score 0 · Answer 2 · answered Jun 18 '23 at 06:43

0

Upgrading to V3 massively increased the size of my Node lambda bundles until I also upgraded the lambda runtime from NODEJS_14_X to NODEJS_18_X.

answered Jun 18 '23 at 06:43

mrwnt10

1,234
1
17
28

score 0 · Answer 3 · answered Jul 19 '23 at 16:19

I also faced this problem and in my case I solved it.

As @siva-sumanth said, you shouldn't include the SDK in your production code. In my case I'm using Serverless with serverless-bundle (a wrapper of serverless-webpack) and it was this package that excluded aws-sdk v2 by default bundling my code...unfortunately it doesn't in v3 because with module package you have to list all the packages you want to exclude (here the issue https://github.com/AnomalyInnovations/serverless-bundle/issues/361)

Example:

plugins:
  - serverless-bundle

custom:
  bundle:
    forceExclude:
      - "@aws-sdk/client-s3"
      - "@aws-sdk/lib-dynamodb"

Nodejs modularized aws-sdk v3 size getting increased

3 Answers3