"Internal Server Error" when querying of DynamoDB in Lambda using Python

Question

I am currently trying to read my dynamodb table "saved_measurements" with the partition key "Name". I'm doing this through API Gateway Lambda proxy integration, and have tested my event['pathParameters']['name'] to be working just fine so that shouldn't be the issue.

However, when I query my dynamodb table with my 'Name', the error appears.

{"message": "Internal server error"}

I have referenced many resources for querying dynamodb but nothing seems to work. I have also tried printing a example string within the response body like "yes" and it works with no issues. Only when I attempt to send my data in my response body do I meet that issue again.

What can I try to resolve this?

import json
import boto3
from boto3.dynamodb.conditions import Key, Attr

def lambda_handler(event, context):
  client = boto3.resource('dynamodb')
  table = client.Table('saved_measurements')

  data = table.query(KeyConditionExpression=Key('Name').eq(event['pathParameters']['name']))
  stuff = data.Items

  response = {
    "statusCode": 200,
    "headers": {
      "Content-Type": 'application/json',
      "Access-Control-Allow-Origin" : "*",
      "Access-Control-Allow-Headers" : "Content-Type",
      "Access-Control-Allow-Methods": "OPTIONS,POST,GET"
    },
    "body": json.dumps(stuff),
    "isBase64Encoded": False
  }
  
  return response

Did you check any logs in CloudWatch for your function for error messages? — Marcin, Oct 01 '22 at 10:15
@Marcin I did not know I could do that. Just checked it, recieved this message ```[ERROR] AttributeError: 'dict' object has no attribute 'Items' Traceback (most recent call last): File "/var/task/lambda_function.py", line 13, in lambda_handler stuff = data.Items``` — beefwhale, Oct 01 '22 at 10:36
OK, so your lambda has crashed. It has a bug in it which is preventing it from executing. — halfer, Oct 01 '22 at 10:49
@halfer I just managed to fix the issue with small adjustments on my end! See the answer below :) Still, thank you everyone for helping me troubleshoot this issue. Really appreciate it. — beefwhale, Oct 01 '22 at 11:05
OK, great stuff. If you are doing any test automation, and you have found a JSON component that can fail in some cases, it may be worth adding a unit test for your build pipeline. That will (hopefully) prevent a regression of the issue. — halfer, Oct 01 '22 at 11:07
The real issue here is that `data` is a dictionary and so `data.Items` is not valid Python. You should use `data["Items"]`. — jarmod, Oct 01 '22 at 12:33

score 0 · Accepted Answer · answered Oct 01 '22 at 10:45

You code needs some optimising, however i'm not certain it will resolve the issue. Let me share how I would write it:

import json
import boto3
from boto3.dynamodb.conditions import Key, Attr
client = boto3.resource('dynamodb')
table = client.Table('saved_measurements')
def lambda_handler(event, context):
    status_code=0
    stuff = ""
    try:
        data = table.query(KeyConditionExpression=Key('Name').eq(event['pathParameters']['name']))
        stuff = json.dumps(data['Items'])
        status_code = 200

    except Exception as e:
        print(e)
        stuff = e
        status_code = 400

    response = {
    "statusCode": status_code,
    "headers": {
        "Content-Type": 'application/json',
        "Access-Control-Allow-Origin" : "*",
        "Access-Control-Allow-Headers" : "Content-Type",
        "Access-Control-Allow-Methods": "OPTIONS,POST,GET"
    },
    "body": stuff,
    "isBase64Encoded": False
    }

    return response

Some points on the code:

Use try/except block anytime you are making API requests to ensure you are not returning exceptions to your downstream tasks unexpectedly
I tend not to use dot notation in Python, hence I changed date.Items to data['Items']. It tends to work more often than dot notation.
Create you clients outside of your request handler in Lambda, this allows Lambda to reuse clients across multiple invocations which will improve your latency.
Don't return 200 for every request regardless on how it resulted, return the correct status code so your downstream processes don't get confused.

Give those changes a go, if you don't succeed, please share your APIGW config.

Your solution and mine returned ```[ERROR] TypeError: Object of type Decimal is not JSON serializable Traceback (most recent call last): File "/var/task/lambda_function.py", line 27, in lambda_handler "body": json.dumps(stuff),``` which is kind of weird considering how my data in dynamoDB only consists of a String Partition key and the rest are whole numbers — beefwhale, Oct 01 '22 at 10:59
Thank you so much for your help, your solution worked for me after I added a line ```import simplejson as json ``` replacing ```import json``` and it worked like magic! Credits: https://stackoverflow.com/a/39257479/19169898 — beefwhale, Oct 01 '22 at 11:02
Your statement about using `data['Items']` rather than `data.items` because it "tends to work more often than dot notation" is highly misleading. Standard Python dicts do not support dot notation at all. You have no choice here. — jarmod, Oct 01 '22 at 12:32
Also, you made a good comment about "not returning exceptions to your downstream tasks unexpectedly" but your exception handler populates `stuff` with the exception and that is returned to the client in the body of the response, which is what you suggested the OP should *not* do. — jarmod, Oct 01 '22 at 12:37
I return the error in `stuff` as it saves adding a conditional check in the Lambda, no where else in this code relies on that variable so the downstream application will be able to parse it just as it would if I had a conditional check in this code. — Leeroy Hannigan, Oct 01 '22 at 17:46

"Internal Server Error" when querying of DynamoDB in Lambda using Python

1 Answers1