AWS Lambda function using pyaudio

Question

My Dream :)

I would like to use pyaudio for a function which will be running on AWS Lambda. However, I get a PythonPipBuilder:ResolveDependencies error when running sam build --use-container

My Setup

I've managed to build my project down to this.

requirements.txt

pyaudio

app.py

def lambda_handler(event, context):
    return {
        'statusCode': 200,
        'body': 'Hello World'
    }

template.yaml (relevant pieces at least)

Resources:
  MyFunction:
    Type: AWS::Serverless::Function 
    Properties:
      CodeUri: my_app/
      Handler: app.lambda_handler
      Runtime: python3.6
      Events:
        MyEvent:
          Type: Api 
          Properties:
            Path: /test
            Method: get

My Problem

When running sam build --use-container --debug I see this output:

Traceback (most recent call last): File "/var/lang/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 42, in execute requirements_path=self.manifest_path, File "/var/lang/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 137, in build_dependencies self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path) File "/var/lang/lib/python3.6/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 201, in build_site_packages raise MissingDependencyError(packages_without_wheels) aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {pyaudio==0.2.11(sdist)}

...

Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - {pyaudio==0.2.11(sdist)}"}}

My Theory

My theory is that this fails because PortAudio is not installed in the Docker container image used when running sam build --use-container

If I run pip install -r requirements.txt on my local machine where I have PortAudio installed, everything works as expected.

My suspicion is that I need to install PortAudio in the AWS Lambda environment somehow...but how? Layers? And if so, from where do I get the binary files I need?

TL;DR;

How do I install PortAudio in an AWS Lambda environment?

I am trying to do the exact same thing - create a Lambda layer for PortAudio. I saw from your comment that you were finally able to do it, but I'm struggling a bit to reproduce your solution. I am trying to do it using the lambci/lambda docker image rather than an EC2 instance, but I think it should translate. I cloned the portaudio repo and built it using make, but now I'm not sure which of the build output files I need, AND I'm not exactly sure where to put the files for my layer zip. libportaudio.so.2.0.0 renamed to libportaudio.so? Copied to /lib in the layer zip? — jamshehan, Sep 08 '20 at 13:24
Hey @jamshehan,you have my sympathy :) I've created a GitHub repo with the dependencies and a vague description. Hope it is useful: https://github.com/pbodsk/aws-lambda-libso-dependencies — pbodsk, Sep 08 '20 at 20:05

score 2 · Accepted Answer · answered Jun 26 '20 at 13:39

2

The quick way to test your lambda is to install the package (e.g pyaudio) in the folder that contains the lambda code, and then zip it all up & upload it. See more info https://docs.aws.amazon.com/lambda/latest/dg/python-package.html#python-package-dependencies

If you have any issue with installing the packages, then it's possible it may be down to the dependence and you may have to add a layer(s).

Once you have your lambda working, you can remove it and have the python package installed using the buildspec.yml. AWS will install the packages when it's running the code pipleine. More details https://docs.aws.amazon.com/lambda/latest/dg/build-pipeline.html

answered Jun 26 '20 at 13:39

Greg

4,468
3
16
26

Thank you for your answer @greg. If I run `pip install --target ./package PyAudio` as suggested in that document, then it works fine on my local machine, which is a Mac. But to my understanding - and I may be wrong - if I upload that, it will fail, because I try to use a .so file from a Mac...which won't work on a Linux machine. It gave me an idea though, so I replicated my setup to an EC2 instance and tried running `pip install --target ./package PyAudio`, but that fails with `.../src/_portaudiomodule.o unable to execute 'gcc': No such file or directory`, so, no portaudiomodule :( – pbodsk Jun 26 '20 at 14:18
If I run `yum search portaudio` I get: `Warning: No matches found for: portaudio` which indicates that for some reason PortAudio is not available out of the box on an AWS Linux installation – pbodsk Jun 26 '20 at 14:19
I think you've answered it, but most packages can be installed using PIP. However PyAudio has to be installed using the wheel (and for the correct OS / version). See https://stackoverflow.com/questions/52283840/i-cant-install-pyaudio-on-windows-how-to-solve-error-microsoft-visual-c-14. For now I would still recommend uploading lambda as a zip (as a basic test). – Greg Jun 26 '20 at 14:35
I tried zipping the files following the description you linked, that resulted in this error: `START RequestId: effa2ea4-f12b-4f5f-91d3-22dcbdf7ad30 Version: $LATEST module initialization error: PortAudio library not found`. I looked into the answer you linked too, but it seems to only mention Windows, or am I missing something. The above may seem like a lot of not particularly uplifting replies but don't get me wrong, I really appreciate you taking the time and effort to help me :) I will go hunting for a Linux Wheel version somewhere. – pbodsk Jun 26 '20 at 16:38
Sorry about that, I'm unable to find the pyaudio wheels for linux. I have had a look at their site https://pypi.org/project/PyAudio/#files. I would expect something with name containing manylinux1_x86_64 e.g: module-name-version-cp36-cp36m-manylinux1_x86_64.whl. Unless someone has created a wheel, you might have to build it. – Greg Jun 26 '20 at 19:17
1

See answers for: https://stackoverflow.com/questions/44538746/cannot-install-pyaudio-0-2-11-in-ubuntu-16-04 should give you an idea how to install it – Greg Jun 26 '20 at 19:24
3

Update. I was finally able to install Portaudio to an EC2 instance (using this: https://aws.amazon.com/premiumsupport/knowledge-center/ec2-enable-epel/) and from there I could install Portaudio and then take the installed so files and add them to a layer which I could add to my function and finally get it working. Man that felt good. I'll accept your answer because you really helped pointing me in the right direction in the comments. Thank you for your time and help, much appreciated :) – pbodsk Jun 28 '20 at 10:54