AWS DynamoDbMapper FilterExpression pagination in Java

Question

I have a dynamoDb table, the table has couple of columns,
My Primary partition key is - pageId (String),
I also have a GSI - pageContainer(String).

I am using DynamoDBMapper to query/scan my table,
I am trying to achieve pagination in DynamoDb,
I understnd paganation in DynamoDB works on ExclusiveStartKey and LastEvaluatedKey.

ExclusiveStartKey will be null on the 1st page query/scan, After querying/scanning the 1st page, DynamoDB will return the LastEvaluatedKey.

If the query result is more than 1Mb or if one has put a limit for the query/scan,
The LastEvaluatedKey's value, is then used as ExclusiveStartKey to query/scan the 2nd page and so on.

My operation requires a Filter Expression beginsWith, on pageContainer column,
as I am trying to get data that beginsWith a certain word, it's like a "Like" condition is compared to relational db.
I have 10 values in the table all start with "Test_test" in the pageContainer column.

Here's my code -

AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard().withEndpointConfiguration(
                new AwsClientBuilder.EndpointConfiguration(DYNAMO_DB_ENDPOINT, DYNAMO_DB_REGION))
                .build();
DynamoDBMapper mapper = new DynamoDBMapper(dynamoDBClient);

String beginWith = "Test_test";
HashMap<String, AttributeValue> filterMap = new HashMap<>();
        filterMap.put(":beginWith", new AttributeValue().withS(beginWith));

DynamoDBScanExpression scanExpression = new DynamoDBScanExpression();   

scanExpression.withFilterExpression("begins_with(pageContainer,:beginWith)")
                .withExpressionAttributeValues(filterMap)
                .withExclusiveStartKey(null)
                .withLimit(4);

ScanResultPage<PageModel> scanPage = mapper.scanPage(PageModel.class, scanExpression);
List<PageModel> pageCmsDomains = scanPage.getResults();

Map<String, AttributeValue> lastEvaluatedKey = scanPage.getLastEvaluatedKey();
System.out.println("LastEvaluatedKey=" + scanPage.getLastEvaluatedKey());

Problem -
The scan operation is returning me the LastEvaluatedKey but, nothing is coming in results(empty response).

I am referring the following resources -
1. https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBMapper.QueryScanExample.html
2. Pagination with DynamoDBMapper Java AWS SDK

did you check my answer? did it work for you? – Alessandro Oliveira Aug 16 '18 at 01:47 — Alessandro Oliveira, Aug 16 '18 at 01:47

score 5 · Accepted Answer · answered Aug 14 '18 at 15:02

The problem you are facing is related to how filter and limit work at DynamoDB.

In DynamoDB the limit seems to be a kind of service protection limiting the amount of records fetched in a single operation, after the data is fetched the filter is applied.

For instance you are using:

scanExpression.withFilterExpression("begins_with(pageContainer,:beginWith)")
                .withExpressionAttributeValues(filterMap)
                .withExclusiveStartKey(null)
                .withLimit(4);

DynamoDB starts fetching the first 4 records, if the records do not match the filter expression, they are removed from response, returning 0 records.

I would say that if you are using filter, you're probably doing something wrong, you'd better create another index and use key filtering.

From AWS DynamoDB Documentation - Working with Queries

Limiting the Number of Items in the Result Set

The Query operation allows you to limit the number of items that it returns in the result. To do this, set the Limit parameter to the maximum number of items that you want.

For example, suppose you Query a table, with a Limit value of 6, and without a filter expression. The Query result will contain the first six items from the table that match the key condition expression from the request.

Now suppose you add a filter expression to the Query. In this case, DynamoDB will apply the filter expression to the six items that were returned, discarding those that do not match. The final Query result will contain 6 items or fewer, depending on the number of items that were filtered.

AWS DynamoDbMapper FilterExpression pagination in Java

1 Answers1