Query More Than 100 DynamoDB Items that match a Partition Key using Java SDK 2?

Question

How can more than 100 items from a dynamodb table, matching a particular partition key, be queried using aws java sdk 2?

If the number of rows that match a hash key exceeds 100 then the batchGetItemPaginator method needs to be used. This method accepts a BatchGetItemRequest which requires a KeysAndAttributes.

When I construct a KeysAndAttributes with just a hash key, but not a sort key, I get the following exception

DynamoDbException: The provided key element does not match the schema

However, if I provide a sort key in the KeysAndAttributes then the batchGetItemPaginator method works fine.

So how can a BatchGetItemRequest be constructed with just a primary key (no sort key)?

Note: my question is not a duplicate of this question. That other question does not involve the required pagination if the number of rows exceeds 100.

Thank you in advance for your consideration and response.

Balu Vyamajala · Accepted Answer · 2021-02-16T00:15:00.970

Primary key >> Partition + HashKey

GetItem is used to get one item by primary key.

BatchGetItem is used to get multiple items from multiple tables using multiple primary keys. so, we need to pass list of Partition & HashKey

We can never do a getItem or batchGetItem with partial key i.e just Partition Key.

To fetch items just by partition key, we must use query api, which returns maximum of 1 MB of data. To fetch all the records with a particular partition key, we can paginate by calling query api multiple times with exclusiveStartKey

.exclusiveStartKey(exclusiveStartKey) helps us to start after a particular record.

.limit(10) usually set to a reasonable number that doesn't exceed total size beyond 1 MB. if no of items is not a concern, we can exclude this.

response.lastEvaluatedKey() capture this from the response to be used for next query.

public static Map<String, AttributeValue> queryTable(DynamoDbClient ddb, String tableName, String partitionKeyName,
        String partitionKeyVal, Map<String, AttributeValue> exclusiveStartKey) {

    Map<String, AttributeValue> lastEvaluatedKey = null;

    HashMap<String, AttributeValue> attrValues = new HashMap<String, AttributeValue>();
    attrValues.put(":" + partitionKeyName, AttributeValue.builder().s(partitionKeyVal).build());
    QueryRequest queryReq = QueryRequest.builder().tableName(tableName).exclusiveStartKey(exclusiveStartKey)
            .keyConditionExpression(partitionKeyName + " = :" + partitionKeyName)
            .expressionAttributeValues(attrValues).limit(10).build();

    try {
        QueryResponse response = ddb.query(queryReq);
        lastEvaluatedKey = response.lastEvaluatedKey();
    } catch (DynamoDbException e) {
        System.err.println(e.getMessage());
        System.exit(1);
    }
    return lastEvaluatedKey;
}

sample test on table test with pk as partition key for value 1

public static void main(String[] args) {

    Region region = Region.US_EAST_1;
    DynamoDbClient ddb = DynamoDbClient.builder().region(region).build();

    Map<String, AttributeValue> lastEvaluatedKey = null;
    // Loop until we don't get lastEvaluatedKey 
    do {
        System.out.println("---------DynamoDb Query with Exclusive Start Key------------" + lastEvaluatedKey);
        lastEvaluatedKey = queryTable(ddb, "test", "pk", "1", lastEvaluatedKey);
    } while (lastEvaluatedKey != null && !lastEvaluatedKey.isEmpty());

    ddb.close();
}

Thank you for the response! The clarification between `query` and `BatchGetItem` is very helpful. However, the documentation for query states that the response is at most 1 MB. Could you please update your example to including pagination? — Ramón J Romero y Vigil, Feb 15 '21 at 19:44
@RamónJRomeroyVigil Updated Answer with paginated query example — Balu Vyamajala, Feb 15 '21 at 20:39

score 0 · Answer 2 · edited Nov 12 '22 at 08:18

0

The accepted answer above is partially wrong (at least as of v2 of the Java SDK), you can do a batch get item with only a primary key.

Unfortunately, the code I have is proprietary therefore I can't share it but just look at the Javadocs around batch get item.

For example: https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/dynamodb/model/BatchGetItemResponse.html

edited Nov 12 '22 at 08:18

flyingfishcattle

1,817
3
14
25

answered Nov 08 '22 at 22:29

Bill Davis

1

Query More Than 100 DynamoDB Items that match a Partition Key using Java SDK 2?

2 Answers2