2

I am trying to perform pagination on a simple DynamoDB table for queries by date range, and had hoped to coerce the LastEvaluatedKey into a "hasMoreResults" boolean value for my front end to consume, but now see that a non-empty LastEvaluatedKey can sometimes appear on results even when the number of items within the date range does not exceed my LIMIT. Does this mean that I will always need to perform a subsequent query that will return no additional items?

My table looks something like:

[
    {PK: "john", SK: "2021-01-01:08:00", amount: 1},
    {PK: "john", SK: "2021-01-01:20:00", amount: 2},
    {PK: "john", SK: "2021-01-02:08:00", amount: 3},
    {PK: "john", SK: "2021-01-02:20:00", amount: 4},
    {PK: "john", SK: "2021-01-03:08:00", amount: 5},
    {PK: "john", SK: "2021-01-03:20:00", amount: 6}
    ...and on for all of January
];

Using the JavaScript DocumentClient, my query looks like:

async function getEntriesByDate({name, startDate, endDate, limit, sort}) {
    return await docClient.query({
        TableName: "someTableName",
        KeyConditionExpression: "#pk = :pk and #sk Between :startDate And :endDate",
        ExpressionAttributeNames: {
            "#pk": "PK",
            "#sk": "SK"
        },
        ExpressionAttributeValues: {
            ":pk": name,
            ":startDate": startDate,
            ":endDate": endDate
        },
        ScanIndexForward: sort === "asc",
        Limit: limit
    }).promise();
}

If I call the function with the end date exactly matching the date of the fourth item and a LIMIT of 4:

getEntriesByDate({name: "john", startDate: "2021-01-01:08:01", endDate: "2021-01-02:20:01", limit: 4, sort:"asc"});

I get the following result:

{
  "Items": [
    {"PK": "john", "SK": "2021-01-01:08:00", "amount": 1},
    {"PK": "john", "SK": "2021-01-01:20:00", "amount": 2},
    {"PK": "john", "SK": "2021-01-02:08:00", "amount": 3},
    {"PK": "john", "SK": "2021-01-02:20:00", "amount": 4 }
  ],
  "Count": 4,
  "ScannedCount": 4
}

Great, there's no LastEvaluatedKey. which is what I expect. But if I call the function with the same args except adding a minute to the end date, I get:

{
  "Items": <same as in last query, which is expected>,
  "Count": 4,
  "ScannedCount": 4,
  "LastEvaluatedKey": {
    "SK": "2021-01-02:20:00",
    "PK": "john"
  }
}

and LastEvaluatedKey does appear, even though there are no additional items that satisfy the query. Is there an idiomatic solution to this problem? Is a subsequent query, perhaps using ExclusiveStartKey, necessary inside my function in order to secure a reliable hasMoreResults value?

TDB
  • 367
  • 1
  • 3
  • 15
  • 1
    I think the issue here is how DynamoDB distributes the data in its partitions, as described in this [documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.Partitions.html#HowItWorks.Partitions.CompositeKey). In short: DynamoDB can not be sure that there are no more items left, that's why it returns the `LastEvaluatedKey` even though no more items are left. But this is a typical problem and I would like to know a good solution for it as well. Looking forward to a great answer :) – Jens Dec 24 '20 at 19:31
  • Thanks for the info. For the time being I've added a second query for the sole purpose of determining whether there are more results. This is for a toy app, so I don't care too much about the inefficiency, but it feels gross, and I hope someone can chime in with a better approach (assuming there is one). – TDB Dec 27 '20 at 01:06

0 Answers0