6

I would like to call the aws s3 sync command from within an AWS Lambda function with a runtime version of Python 3.6. How can I do this?

Why don't you just use the included boto3 SDK?

Architecturally this doesn't make sense!

For my use case I think it makes sense architecturally and financially, but I'm open to alternatives. My Lambda function:

  • downloads Git and Hugo
  • downloads my repository
  • runs Hugo to generate my small (<100 pages) website
  • uploads the generated files to s3

Right now, I'm able to do all of the above on a 1536 MB (the most powerful) Lambda function in around 1-2 seconds. This function is only triggered when I commit changes to my website, so it's inexpensive to run.

Maybe it is already installed in the Lambda environment?

As of the time of this writing, it is not.

Community
  • 1
  • 1
pjgranahan
  • 575
  • 6
  • 15

1 Answers1

5

From Running aws-cli Commands Inside An AWS Lambda Function:

import subprocess
command = ["./aws", "s3", "sync", "--acl", "public-read", "--delete",
           source_dir + "/", "s3://" + to_bucket + "/"]
print(subprocess.check_output(command, stderr=subprocess.STDOUT))

The AWS CLI isn't installed by default on Lambda, so you have to include it in your deployment. Despite running in a Python 3.6 Lambda environment, Python 2.7 is still available in the environment, so the approach outlined in the article will continue to work.

To experiment on Lambda systems, take a look at lambdash.

pjgranahan
  • 575
  • 6
  • 15
John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • 1
    Unfortunately the approach in that post doesn't work in this case. I'll elaborate in my question as to what I've already tried. AWS CLI is still not installed on Python Lambda instances, though I have not tried other environments. – pjgranahan Apr 25 '17 at 06:24
  • 2
    The referenced article shows how to package the AWS CLI with the Lambda function. – John Rotenstein Apr 25 '17 at 08:28
  • Thanks for your help John. Your comment spurred me to revisit the article and I got it working using the blog's approach. I did two key things wrong that others should watch for: 1) I attempted to build aws-cli on Amazon Linux in an effort to match the Lambda environment. This failed with weird PyYaml errors, but building on Ubuntu worked flawlessly. 2) I didn't realize that Python 2.7 was still available on a Python 3.6 Lambda environment, and I attempted to adapt the script given in the blog to Python 3.6. I think this approach could work eventually, but I've abandoned it for now. – pjgranahan Apr 25 '17 at 20:34