23

I'm trying to pip install a package in an AWS Lambda function.

The method recommended by Amazon is to create a zipped deployment package that includes the dependencies and python function all together (as described in AWS Lambda Deployment Package in Python). However, this results in not being able to edit the Lambda function using inline code editing within the AWS Lambda GUI.

So instead, I would like to pip install the package within the AWS Lambda function itself. In AWS Lambda, the filesystem is read-only apart from the /tmp/ directory, so I am trying to pip install to the /tmp/ directory. The function is only called once-daily, so I don't mind about the few extra seconds required to re-pip install the package every time the function is run.

My attempt

def lambda_handler(event, context):
    # pip install dependencies
    print('begin lambda handler')
    import subprocess
    import sys
    subprocess.call('pip install cryptography -t /tmp/ --no-cache-dir'.split())
    from cryptography.fernet import Fernet
    pwd_encrypted = b'gAAAAABeTcT0OXH96ib7TD5-sTII6jMfUXPhMpwWRCF0315rWp4C0yav1XAPIn7prfkkA4tltYiWFAJ22bwuaj0z1CKaGl8vTgNd695SDl25HnLwu1xTzaQ='
    key = b'fP-7YR1hUeVW4KmFmly4JdgotD6qjR52g11RQms6Llo='
    cipher_suite = Fernet(key)
    result = cipher_suite.decrypt(pwd_encrypted).decode('utf-8')
    print(result)
    print('end lambda handler')

However, this results in the error

[ERROR] ModuleNotFoundError: No module named 'cryptography'

I have also tried replacing the subprocess call with the following, as recommended in this stackoverflow answer

    cmd = sys.executable+' -m pip install cryptography -t dependencies --no-cache-dir'
    subprocess.check_call(cmd.split())

However, this results in the error

OSError: [Errno 30] Read-only file system: '/var/task/dependencies'

maurera
  • 1,519
  • 1
  • 15
  • 28
  • 3
    I would not recommend running `pip` inside Lambda. You could package your dependencies into an [AWS Lambda Layer](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html), which would then let you edit the remaining code in the Lambda management console. – John Rotenstein Feb 20 '20 at 02:55
  • @JohnRotenstein - Why is this not recommended? I'm not intending to use this in production. It would make the debugging/learning process easier if everything can be self-contained within a single python file. Are there any blockers to making this work? – maurera Feb 20 '20 at 17:17
  • 1
    Yes, the blockers are exactly what you are experiencing. An AWS Lambda function can only write to the `/tmp/` directory. – John Rotenstein Feb 20 '20 at 20:31
  • @JohnRotenstein - I realize that, and that's why I'm attempting to only write to the /tmp/ directory by specifying /tmp/ as the install target. Is there a way to then import the package, once the package is pip installed to /tmp/? The two methods I've tried of pip-installing to a custom directory seem to work outside of AWS Lambda – maurera Feb 20 '20 at 21:27

2 Answers2

29

I solved this with a one-line adjustment to the original attempt. You just need to add /tmp/ to sys.path so that Python knows to search /tmp/ for the module. All you need to do is add the line sys.path.insert(1, '/tmp/').

Solution

import os
import sys
import subprocess

# pip install custom package to /tmp/ and add to path
subprocess.call('pip install cryptography -t /tmp/ --no-cache-dir'.split(), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
sys.path.insert(1, '/tmp/')
from cryptography.fernet import Fernet

def lambda_handler(event, context):
    # pip install dependencies
    pwd_encrypted = b'gAAAAABeTcT0OXH96ib7TD5-sTII6jMfUXPhMpwWRCF0315rWp4C0yav1XAPIn7prfkkA4tltYiWFAJ22bwuaj0z1CKaGl8vTgNd695SDl25HnLwu1xTzaQ='
    key = b'fP-7YR1hUeVW4KmFmly4JdgotD6qjR52g11RQms6Llo='
    cipher_suite = Fernet(key)
    result = cipher_suite.decrypt(pwd_encrypted).decode('utf-8')
    print(result)

Output

Hello stackoverflow!

Note - as @JohnRotenstein mentioned in the comments, the preferred method to add Python packages is to package dependencies in an AWS Lambda Layer. My solution just shows that it is possible to pip install packages directly in an AWS Lambda function.

maurera
  • 1,519
  • 1
  • 15
  • 28
  • It's not the best solution, because pip installation processes take several seconds to finish which adds up to your bill. – Zenul_Abidin Jun 19 '21 at 20:08
  • 1
    @Zenul_Abidin - you're right. As written in the answer, the preferred method to include a python package is to use an AWS Lambda Layer. However, the question asks whether it's possible to do this within an the Lambda function itself (eg - for testing purposes) – maurera Jun 22 '21 at 15:50
  • 1
    Whenever your lambda function is invoked, then it will try to install library which slows your code down. creating a package with all required libraries is better approach if you have time constraint. (all simply if you dont want to pay much) Check out deployment package with dependencies section in official documentation https://docs.aws.amazon.com/lambda/latest/dg/python-package.html – aykcandem Apr 22 '22 at 14:51
  • I love this hack solution. I hope to never use it, but it's awesome. If your total layer bundle creeps over 250MB, this could help out in a pinch, but then you should obv shift over to container-back Lambdas. In response to the other commenters, no, this won't execute every time your function is invoked. Code outside of the function is only run when the function is *initialised*, which is only the first time it is invoked, i.e., cold start – jameslol Dec 06 '22 at 05:57
  • @jameslol - thanks for that clarification! I hadn't realized that that code only gets executed during a cold start. I just googled about it and also found this reference, which seems to say that cold starts also happen if it's been ~5-7 minutes since the last invocation - https://mikhail.io/serverless/coldstarts/aws/ – maurera Apr 04 '23 at 00:13
  • I believe it's extremely variable. It may be 5-7 minutes, but I would never count on it. You sometimes get a cold start immediately after an invocation, and you sometimes may have a function remain without invocation for much longer – jameslol Apr 14 '23 at 04:04
  • I run a lot of Lambda functions with both high and low traffic. I'd consider cold start optimisation to be job for later. In practice, they're just not that big of a worry. If your traffic is so low that you're getting a high percentage of cold starts, then you probably don't have enough traffic that it's worth worrying. Remember, lots of other things can cause a delay in your function, so I wouldn't lose sleep over this particular thing. You can always optimise this down the track if needed - provisioned concurrency, or by changing the code. Anything else is premature optimisation – jameslol Apr 14 '23 at 04:15
1

For some reason subprocess.call() was returning a FileNotFound error when I was trying to pip3.8 install <package> -t <install-directory>. I solved this by using os.system() instead of subprocess.call(), and I specified the path of pip directly:

os.system('/var/lang/bin/pip3.8 install <package> -t <install-directory>').

Laurie
  • 1,189
  • 1
  • 12
  • 28