0

CloudWatch Log of Lambda Function

I'm using the below code to scan with pagination a dynamodb table to pull 5 records from a maximum of 20 records. When I invoke the api through postman GET method it is pulling the same 5 records each time. Does it mean that my lambda code doesn't work properly or do I need to customize my API call in postman?

import boto3
import os
import json

def lambda_handler(event, context):

    client = boto3.client('dynamodb', region_name='ap-southeast-1')
    pagination_config={
            "MaxItems":20, 
            "PageSize": 5
            }
    
    paginator = client.get_paginator('scan')
    response_iterator = paginator.paginate(
        TableName="Users", 
        PaginationConfig=pagination_config
    )
    for page in response_iterator:
        Items = page['Items']
        print(Items)
        print("--------------------------")

    from botocore.paginate import TokenEncoder
    encoder = TokenEncoder()
    for page in response_iterator:
        if "LastEvaluatedKey" in page:
            encoded_token = encoder.encode({"ExclusiveStartKey": page["LastEvaluatedKey"]})
            pagination_config = {
                    "MaxItems": 20,
                    "PageSize": 5,
                    "StartingToken": encoded_token
                    }
            Items = page['Items']
            
    return {
        'statusCode': 200,
        'headers': {},
        'body': json.dumps(Items)
    }
Sohag Mahamud
  • 1
  • 2

1 Answers1

1

When paginating you need to start with the LastEvaluatedKey of your previous call. In the posted code the same LastEvaluatedKey is being used over and over which will not allow for pagination and will produce the same result. here is an example that you can run locally

import os
import json
from botocore.paginate import TokenEncoder

def lambda_handler(event, context):
    key= event['key'] 
    client = boto3.client('dynamodb', region_name='ap-southeast-1')
    
    if not key:
        pagination_config = {"MaxItems": 20, "PageSize": 5}
    else:
        encoder = TokenEncoder()
        encoded_token=encoder.encode({'ExclusiveStartKey':  {'id': {'S': key}}})
        pagination_config = {"MaxItems": 20, "PageSize": 5, "StartingToken":  encoded_token } 
        

    paginator = client.get_paginator('scan')
    response_iterator = paginator.paginate(
        TableName="Users", 
        PaginationConfig=pagination_config
    )
            
    for page in response_iterator:
        Items = page['Items']
    return {
        'statusCode': 200,
        'headers': {},
        'body': json.dumps(Items),
        'LastEvaluatedKey' : page["LastEvaluatedKey"]
    }

in the response the last evaluated key will look like:

"LastEvaluatedKey": {"id": {"S": "???????"}}}"

be sure to use it for your next pagination.

CruncherBigData
  • 1,112
  • 3
  • 14
  • 34
  • Thank you very much for your answer. I still can't figure out how to call my API with your given code to paginate into my client (Postman). I want to accomplish something like following. There will be a list API where I can see all the entries. Let's assume that my dynamodb table has 200 records. The list API has to be paginated, where I can tell the number of entries I want to get. If the number is not defined then 5 entries will be on the list. I'm trying to learn this things. Is it possible to accomplish this kind of thing using dynamodb pagination feature? – Sohag Mahamud Dec 20 '20 at 15:09
  • 1. I have revised my answer to read the key from the event object. 2. To send a request from postman/curl you need to setup gateway, here is a good example: https://stackoverflow.com/questions/29877220/invoke-a-aws-lambda-function-by-a-http-request – CruncherBigData Dec 20 '20 at 18:57
  • [ERROR] TypeError: lambda_handler() missing 1 required positional argument: 'context' – Sohag Mahamud Dec 20 '20 at 21:09
  • data= lambda_handler("KeyFromLastCall") – Sohag Mahamud Dec 20 '20 at 21:09
  • Thanks again. Maybe being so noob as I'm almost new to the development world. In fact I have already setup api gateway. I tried with your revised code to invoke my api gateway through Postman. It is giving application error. Any thought regarding this error? – Sohag Mahamud Dec 20 '20 at 21:20
  • try again , I removed the code that is meant for testing. Now the code has only the lambda function. – CruncherBigData Dec 21 '20 at 04:30
  • Fixed import issue of boto3. Now giving this error. [ERROR] KeyError: 'key' Traceback (most recent call last): File "/var/task/app.py", line 7, in lambda_handler key= event['key'] – Sohag Mahamud Dec 21 '20 at 07:53
  • in the body of the message your sending via postman/curl you need to provide a key , something like {"key": ""} – CruncherBigData Dec 21 '20 at 18:45
  • I tried with {"key": ""} and {"key": "6d9079ac-5414-4299-a2ff-996c5e621d81"} but both request sends me 403 error to the client. Lambda gives [ERROR] KeyError: 'key' Traceback (most recent call last): File "/var/task/app.py", line 7, in lambda_handler key= event['key'] – Sohag Mahamud Dec 21 '20 at 20:59
  • My dynamodb schema only has primary key which is 'id' as string. my first record is { "datetime": "12/21/2020, 20:45:27", "id": "6d9079ac-5414-4299-a2ff-996c5e621d81", "username": "User 1" } – Sohag Mahamud Dec 21 '20 at 21:01
  • try again I corrected the key extraction. Be sure to have a key item: {"key": ""} – CruncherBigData Dec 22 '20 at 05:32