15

I have a table with close to 2 billion rows already created in DynamoDB.

Due to a query requirement, I had to create a Global Secondary Index(GSI) in it. The process of GSI creation started 36 hours ago but still isn't completed. Portal shows Item Count to be around 100 million. So long way to go.

Questions:

  1. Why does it take such a long time when sufficient WCU and RCU are alotted( 30k in fact ).
  2. GSI partition key I've used is something whose values are repetitive, could that be the reason why GSI creation is taking more time (ideal scenario is that we select a partition key which doesn't repeat for items to span across multiple partitions).
  3. Is there a way to abort the creation of GSI while the process is on? it doesn't allow through AWS console.

Thanks.

Dixit Gokhale
  • 601
  • 1
  • 12
  • 32
  • 2
    Monitor GSI creation using CloudWatch metric: OnlineIndexPercentageProgress. More info here: https://www.abhayachauhan.com/2017/12/dynamodb-global-secondary-indexes – Abhaya Chauhan Dec 21 '17 at 12:05

1 Answers1

13

A GSI has its own WCUs and RCUs, distinct and separate to the primary index. Could this be because you dont have enough WCUs on your GSI?

If your global secondary index is taking too long to create (common when adding indexes on an existing large table), you can provision additional write capacity by following these steps:

Open the DynamoDB console.

From the navigation pane, choose Tables, and then select your table from the list.

Choose the Indexes tab.

Increase the write capacity of the index, and then choose Save.

After about a minute, check the OnlineIndexPercentageProgress metric from the Metrics tab to see if the creation of your global secondary index is progressing satisfactorily.

EDIT: Above from the AWS Knowledge Center

'OnlineIndexPercentageProgress' instructions:

Creation of your global secondary index will begin. You can monitor the progress on the Metrics tab:

Choose the Metrics tab.

Choose View all CloudWatch metrics.

In the CloudWatch console, choose DynamoDB. In the Search Metrics box, enter OnlineIndexPercentageProgress. Note: If the search returns an empty list, wait about a minute for metrics to populate.

Choose the name of the index to see the progress.

Community
  • 1
  • 1
F_SO_K
  • 13,640
  • 5
  • 54
  • 83
  • 1
    I have alotted WCU and RCU to GSI in addition to the table as a whole. – Dixit Gokhale Dec 21 '17 at 09:24
  • For GSI ( 30,000 WCU and RCU ). For table ( 20,000 WCU and RCU ) – Dixit Gokhale Dec 21 '17 at 13:08
  • 2 Billion records at 30k WCUs should take 19 hours, but only if you had no activity on the table. AWS recommend setting your index WCUs to 1.5 times the table WCUs during creation (which you have done). This is to ensure you have capacity to both backfill and maintain the new index concurrently. So if the index was using 20k WCUs to keep up with your table, you'd have 10k WCUs to backfill, which would take 56 hours to clear 2B items, which sounds about right. So I think this just comes down to index WCUs. – F_SO_K Dec 21 '17 at 14:17
  • Also, you can't cancel the index creation. Some details [here](http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.OnlineOps.html#GSI.OnlineOps.Creating) – F_SO_K Dec 21 '17 at 14:18
  • 1
    I was thinking along the same lines. It should not take more than 2-3 days. But it's been already 2 days and value of OnlineIndexPerventage is 7% which is very surprising. – Dixit Gokhale Dec 21 '17 at 16:42
  • Question: if table continues to receive substantial amount of writes per day, does it effect the creation of index? Maybe it is better to stop writes operations while index is being created? Put them in a temporary table and then merge data once index is ready.. – kolodi Oct 31 '18 at 10:32