6
  1. When and how is a Dynamodb GSI having a Partition Key and Sort Key partitioned?
  2. Is there a maximum size limit on GSI Partitions like table partitions?
  3. If yes then what happens when a uni-cardinal GSI (i.e. GSI having the same partition key across all records) exceeds the storage limit?
John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
lalatnayak
  • 160
  • 1
  • 6
  • 21

3 Answers3

3

1) See my answer here https://stackoverflow.com/a/51240423/4985580 for how tables are partitioned. A GSI is essentially just a new table, it is partitioned in the same way as your base table.

https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.Partitions.html

Global secondary indexes in DynamoDB are also composed of partitions. The data in a GSI is stored separately from the data in its base table, but index partitions behave in much the same way as table partitions.

2) Yes, 10GB

3) That's an interesting question and I don't have the answer. Dynamo accesses the correct partition based on the partition key of the data, so if you fill more than one partition with a single partition key, it seems likely you would have a problem. That said you'd probably need at least 2.5 Million items with the same partition key for this to happen (10GB/4KB). Is this a possible scenario for you?

F_SO_K
  • 13,640
  • 5
  • 54
  • 83
  • 1
    Why is the calculation 10GB/4KB? isn't the max size of an item 400KB? in that case it would be only 25000 items to reach the limit of the partition storage, but that would be in the case you are projecting all data into the GSI I think, is that correct? – raspacorp Jan 23 '20 at 19:57
  • 1
    Yes, I think that is correct. I had assumed the smallest item size and of course a table will partition with less items when they are large. So I think 25,000 would be the fewest items to cause a new partition creation.. – F_SO_K Jan 24 '20 at 09:24
  • Great answer @F_SO_K I'm also curious to know a definitive answer for 3. – jellycsc Aug 06 '20 at 03:26
  • 1
    I think (2) is wrong: the 10GB limit only applies to LSI partitions (if I'm not mistaken). GSI and table partitions are limitless. – Lawrence Wagerfield Nov 10 '20 at 00:59
  • You're talking about whether there is a limit on the number if items in a partition (which there isn't). This Q&A is about partitions. Take a look at the answer I linked to and you will find AWS explaining it in detail. – F_SO_K Nov 10 '20 at 09:03
0

If yes then what happens when a uni-cardinal GSI (i.e. GSI having the same partition key across all records) exceeds the storage limit?

DynamoDB partition can only handle 1000 WCUs per sec. Also, DDB only allows 400 WCU per record. If your GSI exceeds WCU the writes to main table also gets throttled. So, if your records is around 400KB, a request rate of around 2.5 can bring your partition down.

Ayan
  • 1
0
  1. Yes, it works in the same way as table partitioning.
  2. Yes, 10GB per partition, also the same as for the table itself.
  3. The behaviour is also exactly the same as for the main table. Dynamo will start spreading the data between partitions using Sorting Key, i.e nothing special happens.

Here is a good blog post that explains how partitioning works: https://aws.amazon.com/blogs/database/choosing-the-right-dynamodb-partition-key/

All items with the same partition key are stored together, and for composite partition keys, are ordered by the sort key value. DynamoDB splits partitions by sort key if the collection size grows bigger than 10 GB.

Sergii Sopin
  • 411
  • 2
  • 11