Boto3 Call to Lambda Function Works in Dev but Not in Production?

Question

I'm trying to launch an AWS Lambda asynchronously from a Django Python app using Boto3. The following code works fine on my local development system:

import boto3
from concurrent.futures import ThreadPoolExecutor

[.....]

endpoint_scoring_arn = "arn:aws:lambda:us-east-1:xxx:function:my-lambda-function"
data = {
    'my_data_param': 'my_data',
}

# https://stackoverflow.com/a/39457165/364966
# https://stackoverflow.com/questions/40377662/boto3-client-noregionerror-you-must-specify-a-region-error-only-sometimes
boto3_client = boto3.client('lambda', region_name='us-east-1')
with ThreadPoolExecutor(max_workers=1) as executor:
future_execution = [executor.submit(boto3_client.invoke,
                FunctionName=endpoint_scoring_arn,
                InvocationType="Event",
                Payload=json.dumps(data)
                )]
try:
    print("boto3 exception:")
    print(future_execution[0]._exception.kwargs['report'])
except Exception as e0:
    print("No boto3 exception found")

...but when I run it in production on EC2, the Lambda function never gets run. There are no error messages in my Django debug.log, or in syslog, and the above code prints out "No boto3 exception found".

What could account for this?

I'm not familiar with the concurrent.futures library. Is that the typical way to deal with potential exceptions? Seems very odd. I'm assuming from your answer below that you actually were triggering an AccessDenied exception from boto3. The fact that you didn't see this clearly in the output, and you consequently found the problem difficult to debug, seems to suggest that you're not dealing with exceptions optimally. — jarmod, Dec 19 '19 at 03:18
Thanks for this info. I used concurrent.futures based on the SO post noted in the code sample posted. I also thought it was odd that I wasn't seeing an AccessDenied exception. If convenient, please post your preferred sample code for future reference. — VikR, Dec 19 '19 at 17:40

score 1 · Answer 1 · answered Dec 19 '19 at 18:08

You're mishandling the exception being raised as a result of providing invalid, or no, credentials.

You can see the exception more clearly as follows:

with ThreadPoolExecutor(max_workers=1) as executor:
    future = executor.submit(
        client.invoke,
        FunctionName=lambda_arn,
        InvocationType="Event",
        Payload=json.dumps(data)
    )
    try:
        print("executor boto3 result:", future.result())
    except Exception as exc:
        print("executor exception:", exc.response)
        print("executor exception code:", exc.response['Error']['Code'])

This will print out, among other things:

executor exception code: AccessDeniedException

score 0 · Accepted Answer · answered Dec 18 '19 at 19:30

The answer turned out to be installing my AWS credentials on EC2:

https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html

This is another approach:

https://aws.amazon.com/blogs/security/a-safer-way-to-distribute-aws-credentials-to-ec2/

Boto3 Call to Lambda Function Works in Dev but Not in Production?

2 Answers2