19

I am now working on a web app associated with Amazon DynamoDB, I want to achieve a function that my users can directly get to the Nth page to view the item info,

I have been told that the pagination in DynamoDB is based on last key, rather than limit/offset.It doesn't natively support offset.DynamoDB Scan / Query Pagination

Does that mean : If I want to get to the 10th page of items, then I have to query the 9 pages ahead first?(which seems reeeeeally not a good solution)

Is there a easier way to do that?

Wenhan Du
  • 372
  • 1
  • 3
  • 10

1 Answers1

32

You are right. DynamoDB doesn't support numerical offset. The only way to paginate is to use the LastEvaluatedKey parameter when making a request. You still have some good options to achieve pagination using a number.

Fast Cursor

You can make fast pagination requests by discarding the full result and getting only the Keys. You are limited to 1MB per request. This represents a large amount of Keys! Using this, you can move your cursor to the required position and start reading full objects.

This solution is acceptable for small/medium datasets. You will run into performance and cost issues on large datasets.

Numerical index

You can also create a global secondary index where you will paginate your dataset. You can add for example an offset property to all your objects. You can query this global index directly to get the desired page.

Obviously this only works if you don't use any custom filter... And you have to maintain this value when inserting/deleting/updating objects. So this solution is only good if you have an 'append only' dataset

Cached Cursor

This solution is built on the first one. But instead of fetching keys every single time, you can cache the pages positions and reuse them for other requests. Cache tools like redis or memcached can help you to achieve that.

  1. You check the cache to see if pages are already calculated
  2. If not, you scan your dataset getting only Keys. Then you store the starting Key of each page in your cache.
  3. You request the desired page to fetch full objects

Choose the solution that fits your needs. I hope this will help you :)

Sébastien
  • 1,015
  • 8
  • 7
  • 10
    using the Fast Cursor method, if I have 10,000 items and want to read the final 100 items, do I still first need to scan the first 9900 items? This is not cost effective. With DynamoDB, even if you do not return the scanned items in the response you are still charged for the items scanned. – Lloyd Oct 18 '18 at 21:57
  • An alternative implementation to "Cached Cursor" is to push state back to the caller by sending the data to them (usually by obfuscating first) and requiring that they send the data back on subsequent calls. – Matt Klein Nov 07 '18 at 16:23
  • Alternatively, stream your DynamoDB data into S3, configure an External Glue table and use Athena to query your data. – DR. May 06 '19 at 08:58
  • Fast cursor method is not actually viable. Projection and filter expressions only occur after all data is retrieved, so the 1MB limit applies to all of the data in the partition, not just the atttribute you are querying. – PhillyTheThrilly Apr 29 '20 at 16:35
  • 1
    I haven't worked with dynamodb for quite a long time now so maybe my response isn't valid anymore. But if I remember well, to be able to stay in the 1MB limit, you need to create a global secondary index that contains only what you want to filter one. – Sébastien Apr 30 '20 at 15:23
  • @Sébastien DynamoDb hasn't changed in that aspect, so you're right. – Jose A Aug 02 '20 at 15:45
  • @DR. Athena is not suitable for OLTP queries. – jellycsc Aug 07 '20 at 02:39