How to Have NaCL at AWS Lambda Properly Working?

Question

I am using Pychromeless repo with success at AWS lambda.

Now I need to use NaCL dependency, to decrypt a string, but I am getting

Unable to import module 'lambda_function': /var/task/lib/nacl/_sodium.abi3.so

with a complement of

invalid ELF header

when running the function on AWS Lambda.

I know it is a problem specific related to AWS Lambda environment, because I can run the function on my Mac inside docker.

Here's my requirements.txt file

boto3==1.6.18
botocore==1.9.18
selenium==2.53.6
chromedriver-install==1.0.3
beautifulsoup4==4.6.1
certifi==2018.11.29
chardet==3.0.4
editdistance==0.5.3
future==0.17.1
idna==2.7
python-telegram-bot==10.1.0
requests==2.19.1
soupsieve==1.7.3
urllib3==1.23
PyNaCl==1.3.0

Here is the dockerfile

FROM lambci/lambda:python3.6
MAINTAINER tech@21buttons.com

USER root

ENV APP_DIR /var/task

WORKDIR $APP_DIR

COPY requirements.txt .
COPY bin ./bin
COPY lib ./lib

RUN mkdir -p $APP_DIR/lib
RUN pip3 install -r requirements.txt -t /var/task/lib

And the makefile:

clean:
    rm -rf build build.zip
    rm -rf __pycache__

fetch-dependencies:
    mkdir -p bin/

    # Get chromedriver
    curl -SL https://chromedriver.storage.googleapis.com/2.37/chromedriver_linux64.zip > chromedriver.zip
    unzip chromedriver.zip -d bin/

    # Get Headless-chrome
    curl -SL https://github.com/adieuadieu/serverless-chrome/releases/download/v1.0.0-37/stable-headless-chromium-amazonlinux-2017-03.zip > headless-chromium.zip
    unzip headless-chromium.zip -d bin/

    # Clean
    rm headless-chromium.zip chromedriver.zip

docker-build:
    docker-compose build

docker-run:
    docker-compose run lambda src/lambda_function.lambda_handler


build-lambda-package: clean fetch-dependencies
    mkdir build
    cp -r src build/.
    cp -r bin build/.
    cp -r lib build/.
    pip install -r requirements.txt -t build/lib/.
    cd build; zip -9qr build.zip .
    cp build/build.zip .
    rm -rf build

Without the decryption part, the code works great. So the issue is 100% related to PyNaCl.

Any help on solving this?

So fast? `numpy/core/multiarray.so` and `/lib/nacl/_sodium.abi3.so` are two different things. The only similarity is the error thrown by lambda: `invalid ELF header`. Please, don't invalidate my question, man. — Luis Rock, Sep 09 '19 at 11:28
Well, I am using docker. So, I was expecting some useful tips/lines for my dockerfile. If I need to "spin up an ec2 instance and create virtualenv with the necessary dependencies", what would be the use of docker? — Luis Rock, Sep 09 '19 at 11:35

LinPy · Answer 1 · 2019-09-09T17:40:42.970

0

I think you may try to setup PyNaCl like so:

SODIUM_INSTALL=system pip3 install pynacl

that will force PyNaCl to use the version of libsodium provided by AWS

see this

in the last version of PyNaClis updated to libsodium 1.0.16. so maybe it is not compatible with AWS

so you may remove PyNaCl from requirements.txt and add this to your Dockerfile:

RUN SODIUM_INSTALL=system pip3 install pynacl -t /var/task/lib

or maybe setup the dockerfile like this and keep PyNaCl in requirements.txt:

ARG SODIUM_INSTALL=system

Try also to setup sodium before installing PyNaCI:

RUN wget https://download.libsodium.org/libsodium/releases/libsodium-1.0.15.tar.gz \ && tar xzf libsodium-1.0.15.tar.gz \ && cd libsodium-1.0.15 \ && ./configure \ && make install

edited Sep 09 '19 at 17:40

answered Sep 09 '19 at 12:18

LinPy

16,987
4
43
57

Interesting. So where does this `SODIUM_INSTALL=system pip install pynacl` line go? – Luis Rock Sep 09 '19 at 12:28
Done. Now the error has changed. AWS Lambda says: `Unable to import module 'lambda_function': No module named 'nacl'`. I must say 2 things: 1) I have `import nacl.secret` and `import nacl.encoding` on my lambda function, 2) Everything worked OK running docker on my machine. – Luis Rock Sep 09 '19 at 13:00
ah sorry you need to use pip3 like the other modules – LinPy Sep 09 '19 at 13:05
All options worked fine running docker on my machine. None of the options worked on AWS Lambda: with `pynacl` on "requirements.txt" = `Unable to import module 'lambda_function': /var/task/lib/nacl/_sodium.abi3.so: invalid ELF header`. Without it = `Unable to import module 'lambda_function': No module named 'nacl'`. *dockerfile was set up like you said. – Luis Rock Sep 09 '19 at 13:42
See the update , now I am out of ideas :) Please Post tue Solution if you find it – LinPy Sep 09 '19 at 14:57
`/bin/sh: wget: command not found` Thanks for your help and time. I will try other ways. – Luis Rock Sep 10 '19 at 09:19

score 0 · Answer 2 · answered Sep 10 '19 at 15:45

Ok, this is how I did it. I had to build everything on an EC2 AMI Linux 2 instance.

amzn2-ami-hvm-2.0.20190823.1-x86_64-gp2 (ami-0a1f49a762473adbd)

After launching the instance, I use this script to install Python 3.6 (and pip) and to create and activate a virtual environment.

For the docker part, I followed this tutorial, not without some troubles on the way (had to

sudo yum install polkit

and

sudo usermod -a -G docker ec2-user

and

reboot

and

sudo curl -L https://github.com/docker/compose/releases/download/1.22.0/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose

and

sudo chmod +x /usr/local/bin/docker-compose).

But anyway, I manage to work with docker on the EC2 instance, building the zip file and uploading it to Lambda environment, where everything worked fine, as I expected.

I thought docker was an environment independent from the host, but I guess that is not the case.

How to Have NaCL at AWS Lambda Properly Working?

2 Answers2