Unable to import lxml etree on aws lambda

Question

{
  "errorMessage": "Unable to import module 'lambda_function': 
        cannot import name   'etree' from 'lxml' (/var/task/lxml/__init__.py)",
  "errorType": "Runtime.ImportModuleError"
}

Also tried https://gist.github.com/allen-munsch/ad8faf9c04b72aa8d0808fa8953bc639:

{
  "errorMessage": "Unable to import module 'lambda_function': 
     cannot import name 'etree' from 'lxml' 
     (/var/task/lxml-4.3.4-py3.6-linux-x86_64.egg/lxml/__init__.py)",
  "errorType": "Runtime.ImportModuleError"
}

I am running on Ubuntu 18.04 on my local machine, and have also tried using the "Amazon Linux" image on an ec2 instance to build the bundle.

I have also tried, while within the activated venv:

STATIC_DEPS=true pip3 install lxml --target ./package --upgrade --no-cache-dir

I have also tried copying the shared object files based on pulling which files were opened when running the script with strace:

#! /bin/bash

export Z=$(pwd)/ok-daily-lambda.zip
rm $Z
zip $Z lambda_function.py
zip $Z __init__.py

for dir in $(find venv_here/lib/python3.6/site-packages)
do
    if [ -d $dir ] ; then
        pushd $dir
        echo zip -r9 $Z $(pwd)
        zip -r9 $Z $(pwd)
        popd
    fi
done

export LIBD=$(pwd)/lib
mkdir -p $LIBD

cp "/home/jmunsch/.local/lib/python3.6/site-packages/.libs_cffi_backend/libffi-d78936b1.so.6.0.4" $LIBD
cp "/lib/x86_64-linux-gnu/libbz2.so.1.0" $LIBD
cp "/lib/x86_64-linux-gnu/libc.so.6" $LIBD
cp "/lib/x86_64-linux-gnu/libdl.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libexpat.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libgcc_s.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/liblzma.so.5" $LIBD
cp "/lib/x86_64-linux-gnu/libm.so.6" $LIBD
cp "/lib/x86_64-linux-gnu/libnss_dns.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libnss_files.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libnss_mdns4_minimal.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/libpthread.so.0" $LIBD
cp "/lib/x86_64-linux-gnu/libresolv.so.2" $LIBD
cp "/lib/x86_64-linux-gnu/librt.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libtinfo.so.5" $LIBD
cp "/lib/x86_64-linux-gnu/libudev.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libutil.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libuuid.so.1" $LIBD
cp "/lib/x86_64-linux-gnu/libz.so.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libapt-pkg.so.5.0" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libffi.so.6" $LIBD
cp "/usr/lib/x86_64-linux-gnu/liblz4.so.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libmpdec.so.2" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libssl.so.1.1" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libstdc++.so.6" $LIBD
cp "/usr/lib/x86_64-linux-gnu/libzstd.so.1" $LIBD

zip -r $Z $LIBD

AWS_ACCESS_KEY_ID="xxx" AWS_SECRET_ACCESS_KEY="xxx" AWS_DEFAULT_REGION="us-east-1" aws lambda update-function-code --function-name ok-today --zip-file fileb://ok-daily-lambda.zip

Here is the directory structure of the most recent zip file:

https://gist.github.com/allen-munsch/5ddc27873e64db6ff106ab828febf434

Shubham Nigam · Answer 1 · 2020-05-15T08:51:02.133

10

I came across a similar problem and I figured out one quick workaround

Using pre-compiled build of lxml

Download https://github.com/shubh2502/aws-lambda-lxml

Folder 3.6.4 and 3.8.0 are lxml versions
Inside lxml there are two builds python27 and python36
As per AWS Lambda python version choose either one of them
Inside python27 and python36 there is lxml folder
Zip code with lxml folder and make sure python has the same version
In Case of AWS Lambda layer put lxml folder into this structure -

python/lib/python3.6/site-packages

I spent lots of time in docker and building these stuff, this method was savior for me, I hope this will help you out

edited May 15 '20 at 08:51

answered May 14 '20 at 05:56

Shubham Nigam

3,844
19
32

I was creating zip as lambda.zip, not sure if lxml was the issue or the name of the zip – aniruddha Nov 03 '20 at 18:09
This didn't work for me, I put the lxml folder inside of my code folder and zipped everything together and still receive the same error – ReactNewbie123 Feb 27 '23 at 18:02

jslipknot · Accepted Answer · 2019-06-29T21:59:09.173

There are modules that cannot be added directly into the site-packages directory to be recognised inside an AWS Lambda environment. When that happens, you have to get an Amazon Linux image from Docker repositories and make your own compiled environment in a container version that will run on AWS Lambda

For example, if you want to use Python 3.6 a good choice will be amazonlinux:2018.03 in case you want to install more packages e.g. pandas, numpy, scipy

    docker run -v $(pwd):/outputs -it amazonlinux:2018.03

Since Amazon Linux is based on Red Hat, you have to install via yum all dependencies when running docker and having already created your virtual environment

    yum update -y
    yum install -y \
      python36 \
      python36-devel \
      python36-virtualenv \
      python34-setuptools \
      gcc \
      gcc-c++ \
      findutils \
      rsync \
      Cython \
      findutils \
      which \
      gzip \
      tar \
      man-pages \
      man \
      wget \
      make \
      zip

For lxml you will also need

    (lambda_docker) bash-4.2# yum install libxml2
    ...
    (lambda_docker) bash-4.2# yum install libxslt
    ...

You install the module as usual

    pip3.6 install lxml

You should see something like

    (lambda_docker) bash-4.2# pip3.6 install lxml
    Collecting lxml
      Downloading https://files.pythonhosted.org/packages/2d/53/34a9f0c79c548e430148837892b6ae91adee571a0e8b6c17bd7ff9c2d12e/lxml-4.3.4-cp36-cp36m-manylinux1_x86_64.whl (5.7MB)
         |################################| 5.7MB 2.0MB/s 
    Installing collected packages: lxml

Then, create your function lambda_function.py and add it in the zipped package pushd-ed and popd-ed from you docker session

    from lxml import etree

    def lambda_handler(event, context):
        print(__name__)
        print(etree.LXML_VERSION)

After created

    zip -g site-packages.zip lambda_function.py

Before uploading, you can check that your zip file contains the lxml directories

    [jonathan@docker lambda_docker]$ unzip -l site-packages.zip 
    Archive:  site-packages.zip
      Length      Date    Time    Name
    ---------  ---------- -----   ----
            0  06-29-2019 23:09   __pycache__/
          277  06-29-2019 23:09   __pycache__/easy_install.cpython-36.pyc
          126  06-29-2019 23:09   easy_install.py
          119  06-29-2019 23:29   lambda_function.py
            0  06-29-2019 23:21   lib/
            0  06-29-2019 23:39   lxml/
            0  06-29-2019 23:37   lxml-4.3.4.dist-info/
            4  06-29-2019 23:37   lxml-4.3.4.dist-info/INSTALLER
         2954  06-29-2019 23:37   lxml-4.3.4.dist-info/METADATA
        13384  06-29-2019 23:37   lxml-4.3.4.dist-info/RECORD
          109  06-29-2019 23:37   lxml-4.3.4.dist-info/WHEEL
            5  06-29-2019 23:37   lxml-4.3.4.dist-info/top_level.txt
         7668  06-29-2019 23:37   lxml/ElementInclude.py
          551  06-29-2019 23:37   lxml/__init__.py
            0  06-29-2019 23:37   lxml/__pycache__/
         3331  06-29-2019 23:37   lxml/__pycache__/ElementInclude.cpython-36.pyc
    ...

Now upload the zip and create an empty test {} in your lambda function

Result

    START RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64 Version: $LATEST
    lambda_function
    (4, 3, 4, 0)
    END RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64
    REPORT RequestId: bb240a17-c2dd-4d63-92c8-fa7561c09f64  Duration: 0.30 ms   Billed Duration: 100 ms     Memory Size: 128 MB Max Memory Used: 50 MB

If you prefer an image

Perfectly ready for AWS Lambda

Hope it helps (:

"there are modules that cannot be added directly into the site-packages" - why is this? Is this a bug? I'm trying to use a simple `BeautifulSoup(site.text, 'lxml')` and it seems overkill to create a container image. — Bn.F76, Aug 29 '22 at 23:12

score 4 · Answer 3 · answered Feb 01 '21 at 13:06

Reason:

The .so files compiled on a different OS. (In my case macOS.)

Solution:

The following article was very helpful:

How do I create a Lambda layer using a simulated Lambda environment with Docker?

All I had to do is:

cd /path_to_requirements_txt/

docker run -v "$PWD":/var/task "lambci/lambda:build-python3.7" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.7/site-packages/; exit"

It created python/lib/python3.7/site-packages/ with all the dependencies!

Bharat · Answer 4 · 2021-04-27T09:01:18.763

3

If you using SERVERLESS for deployment:

Then, check the python version on your local environment as well as the lambda runtime version you are using

I was using Python 3.8 on my local and the lambda runtime version was using was Python 3.7 So I changed the lambda version to use Python 3.8

THAT WORKED FOR ME!!!

I hope this will help you out

edited Apr 27 '21 at 09:01

answered Apr 26 '21 at 13:42

Bharat

201
2
9

Unable to import lxml etree on aws lambda

4 Answers4

Reason:

Solution:

Linked