AWS nodejs lambda layer: cannot open shared object file: No such file or directory

Question

I am developing a lambda-based API in Node.js and deploying it through Terraform. As my development machine is Mac OS X, and i believe that the lambda execution environment is something like: OS/Arch: Linux, x86-64, ARM 64 (anyway, its not Mac OS X), the word is that I need to compile my non-AWS dependencies in a similar OS and use a Lambda layer to house them.

I won't go into the gory Terraform details, suffice it to say that in the AWS Lambda console you can see that there is an associated Lambda layer:

the layer is a zip file and both it and the lambda are running in Node.js 18.x

The structure of the zip file itself is:

% unzip -l zgf-layer-2023-04-08T19:34:20Z.zip
Archive:  zgf-layer-2023-04-08T19:34:20Z.zip
Length      Date    Time    Name
---------  ---------- -----   ----
    0  04-09-2023 05:35   nodejs/
    0  04-09-2023 05:35   nodejs/node_modules/
 7374  04-09-2023 05:34   nodejs/node_modules/http-status-codes/README.md
    0  04-09-2023 05:34   nodejs/node_modules/http-status-codes/build/
    0  04-09-2023 05:34   nodejs/node_modules/@types/bcrypt/
  695  04-09-2023 05:34   nodejs/node_modules/@types/bcrypt/README.md
 7187  04-09-2023 05:34   nodejs/node_modules/@types/bcrypt/index.d.ts
    0  04-09-2023 05:34   nodejs/node_modules/@types/jsonwebtoken/
 1079  04-09-2023 05:34   nodejs/node_modules/@types/jsonwebtoken/README.md
 9349  04-09-2023 05:34   nodejs/node_modules/@types/jsonwebtoken/index.d.ts
    0  04-09-2023 05:34   nodejs/node_modules/jsonwebtoken/
    0  04-09-2023 05:34   nodejs/node_modules/uuidv4/build/
 1086  04-09-2023 05:34   nodejs/node_modules/uuidv4/LICENSE.txt
    0  04-09-2023 05:34   nodejs/node_modules/jws/
    0  04-09-2023 05:34   nodejs/node_modules/http-status-codes/
 1310  04-09-2023 05:34   nodejs/node_modules/http-status-codes/package.json
    0  04-09-2023 05:34   nodejs/node_modules/@types/
 1141  04-09-2023 05:34   nodejs/node_modules/@types/bcrypt/LICENSE
 1193  04-09-2023 05:34   nodejs/node_modules/@types/bcrypt/package.json
 1141  04-09-2023 05:34   nodejs/node_modules/@types/jsonwebtoken/LICENSE
 2488  04-09-2023 05:34   nodejs/node_modules/@types/jsonwebtoken/package.json
 1148  04-09-2023 05:34   nodejs/node_modules/bcrypt/promises.js
  284  04-09-2023 05:34   nodejs/node_modules/bcrypt/.editorconfig
  221  04-09-2023 05:34   nodejs/node_modules/bcrypt/test_alpine.sh
 1026  04-09-2023 05:34   nodejs/node_modules/bcrypt/ISSUE_TEMPLATE.md
 2502  04-09-2023 05:34   nodejs/node_modules/bcrypt/package.json
  857  04-09-2023 05:34   nodejs/node_modules/bcrypt/examples/async_compare.js
  165  04-09-2023 05:34   nodejs/node_modules/bcrypt/examples/forever_gen_salt.js
    0  04-09-2023 05:34   nodejs/node_modules/bcrypt/.github/workflows/
  918  04-09-2023 05:34   nodejs/node_modules/bcrypt/.github/workflows/ci.yaml
    0  04-09-2023 05:34   nodejs/node_modules/bcrypt/lib/binding/
    0  04-09-2023 05:35   nodejs/node_modules/bcrypt/lib/binding/napi-v3/
48536  10-13-2022 22:23   nodejs/node_modules/bcrypt/lib/binding/napi-v3/bcrypt_lib.node
  474  04-09-2023 05:34   nodejs/node_modules/bcrypt/SECURITY.md
10273  04-09-2023 05:34   nodejs/node_modules/bcrypt/src/bcrypt_node.cc
 4772  04-09-2023 05:34   nodejs/node_modules/bcrypt/src/node_blf.h
  428  04-09-2023 05:34   nodejs/node_modules/jsonwebtoken/lib/JsonWebTokenError.js
  107  04-09-2023 05:34   nodejs/node_modules/jsonwebtoken/lib/psSupported.js
  395  04-09-2023 05:34   nodejs/node_modules/jsonwebtoken/lib/TokenExpiredError.js
    0  04-09-2023 05:34   nodejs/node_modules/uuidv4/
    0  04-09-2023 05:34   nodejs/node_modules/uuidv4/lib/
 1223  04-09-2023 05:34   nodejs/node_modules/uuidv4/lib/uuidv4.ts
  323  04-09-2023 05:34   nodejs/node_modules/uuidv4/tsconfig.json
    0  04-09-2023 05:34   nodejs/node_modules/uuidv4/build/lib/
 1500  04-09-2023 05:34   nodejs/node_modules/uuidv4/build/lib/uuidv4.js
  483  04-09-2023 05:34   nodejs/node_modules/uuidv4/build/lib/uuidv4.d.ts
 ---------                     -------
 47442776                     2109 files

I have chopped a lot out of this, but for the error below, note the line that says:

nodejs/node_modules/bcrypt/lib/binding/napi-v3/bcrypt_lib.node

and now the error (reported through cloudwatch)

2023-04-08T19:36:43.629Z    undefined   ERROR   Uncaught Exception  
{
    "errorType": "Error",
    "errorMessage": "/opt/nodejs/node_modules/bcrypt/lib/binding/napi-v3/bcrypt_lib.node: cannot open shared object file: No such file or directory",
    "code": "ERR_DLOPEN_FAILED",
    "stack": [
        "Error: /opt/nodejs/node_modules/bcrypt/lib/binding/napi-v3/bcrypt_lib.node: cannot open shared object file: No such file or directory",
        "    at Module._extensions..node (node:internal/modules/cjs/loader:1302:18)",
        "    at Module.load (node:internal/modules/cjs/loader:1081:32)",
        "    at Module._load (node:internal/modules/cjs/loader:922:12)",
        "    at Module.require (node:internal/modules/cjs/loader:1105:19)",
        "    at require (node:internal/modules/cjs/helpers:103:18)",
        "    at Object.<anonymous> (/opt/nodejs/node_modules/bcrypt/bcrypt.js:6:16)",
        "    at Module._compile (node:internal/modules/cjs/loader:1218:14)",
        "    at Module._extensions..js (node:internal/modules/cjs/loader:1272:10)",
        "    at Module.load (node:internal/modules/cjs/loader:1081:32)",
        "    at Module._load (node:internal/modules/cjs/loader:922:12)"
    ]
}

My question is: why can't this be found?

According to the docos I've read so far. The lambda Layer should be composed of a zipped archive, which contains nodejs as its root directory, and inside that: a directory called node_modules. I did the usual npm init and imported (in this case) the bcrypt library through npm i bcrypt. I did all this inside a running docker container with the AWS lambda image documented here, so that the compilations would be done in linux.

The only thing I can think of is maybe I need /opt in the path before nodejs? none of the doco says I need to do that, in fact most of it is pretty emphatic about having nothing other than nodejs at the root.

What am I missing?

Your ZIP file hierarchy seems correct to me. Double-check the Amazon Linux 2 build of bcrypt? Also be aware of a [native JS implementation of bcrypt](https://github.com/dcodeIO/bcrypt.js). — jarmod, Apr 08 '23 at 21:50
hmmm...errors in the EOF (OS) or syntax (transpilation) are what brought me to compiling in AWS Linux (on Docker). I feel that if either of these were the issue, then I'd get these errors again, not a total failure to even find the file... anyway, will check — Michael Coxon, Apr 08 '23 at 22:34
Agree that it's unlikely cause as that'd probably result in ERR_DLOPEN_FAILED with some other underlying cause such as invalid ELF header. You could also log the content of the /opt/nodejs/node_modules/bcrypt/ folder to verify it's what you expected after the Lambda Layer is unzipped. — jarmod, Apr 08 '23 at 22:56
Related: https://stackoverflow.com/questions/71027495/i-received-this-code-error-each-time-i-tried-to-deploy-may-app-on-herokucode and https://stackoverflow.com/questions/43716236/my-code-doesnt-work-when-i-deploy-it-to-heroku — jarmod, Apr 08 '23 at 23:02
Ok so I had some time to try out changing bcrypt for bcryptjs, and, well, it worked. So thank you. I wish I knew why it worked. Now I wonder if all the other libs I intend to use are going to work in the lambda environment. Since all of this worked locally, I can't reliably test it during build, only after deployment. — Michael Coxon, Apr 09 '23 at 01:14
It certainly seems like it should work if you've deployed the correct, native Amazon Linux 2-compatible pre-requisites. If you want to pursue it, I'd be temped to debug this on an Amazon Linux 2 EC2 instance rather than Lambda initially. — jarmod, Apr 09 '23 at 16:59

AWS nodejs lambda layer: cannot open shared object file: No such file or directory

0 Answers0