3

I have a lambda function runs on Python 3.7 with architecture x86_64 before. Now I would like to migrate it to arm64 to use the Graviton processor and upgrade to Python 3.9 as well.

While I success to create the Python 3.9 virtual environment layer with the dependencies that I need, which is aws-encryption-sdk, when I change the architecture of my lambda function to arm64 and runtime to Python 3.9, below error shows after I test my code:

Unable to import module 'encryptor-lambda': /opt/python/cryptography/hazmat/bindings/_rust.abi3.so: cannot open shared object file: No such file or directory",

I went and check my virtual env layer and pretty sure the file /opt/python/cryptography/hazmat/bindings/_rust.abi3.so is existed there.

Then I tried to keep my runtime at Python 3.9 and switched back the architecture to x86, it works! Only if I try to change it to arm64, it has that error above.

I look up online and can't seems to have a solution or as of why is that. Is it not possible to migrate for the lambda functions that requires dependencies? Or am I missing anything?

Muhammad Dyas Yaskur
  • 6,914
  • 10
  • 48
  • 73
Sivvie Lim
  • 784
  • 2
  • 14
  • 43

2 Answers2

4

Libraries like aws-encryption-sdk-python sometimes contain code/dependencies that are not pure Python and need to be compiled. When code needs to be "compiled" it is usually compiled for a target architecture (like ARM or x86) to run properly.

You can not run code compiled for one architecture on different architecture. So I suspect that is the reason for your error.

Looking at the error message I suspect it is the cryptography library causing this issue.

The library uses Rust. If you check your error, you will see that the shared library for the Rust binding is the causing your error (_rust.abi3.so). According to the documentation of the library, the ARM architecture is supported.

Therefore, I suspect that the way you are packaging your Lambda deployment package and it's dependencies is the issue. You are probably doing that on a computer with x86 architecture. Package manager like pip usually detect the OS and architecture they are run on and download dependencies for those OS's and architectures.

So I guess you have two options:

  1. Run your build/deployment on an ARM machine
  2. Somehow manage to "cross compile" with tools like crossenv

Both options are not really great.

Unfortunately, this is one of those areas where Python Lambdas can become very cumbersome to develop/deploy. Every time a depdency uses a non-Python extension (like a C extension), packaging/deployment becomes a problem.

Maybe someone else has a great tool to recommend.

Jens
  • 20,533
  • 11
  • 60
  • 86
  • I see, then maybe I will have to halt this process first until I can find a better solution. Thank you for the information! – Sivvie Lim Jan 14 '22 at 02:28
0

The answer that Jens gave is correct. But more concisely, you built your environment on x86_64 and downloaded the x86_64 wheel for cryptography. You will need to build the environment on Arm to deploy on Arm.

Another option you could consider is using Docker buildx to build for both platforms under emulation with qemu, as long as the build doesn't take long. Emulation slows things down significantly.

Jonathan Swinney
  • 1,054
  • 9
  • 29