4

I have a DynamoDB table configured with auto-scaling enabled. I also have a Lambda function that is writing to this table. I've started seeing throttling exceptions in the lambda and the table also reports lots of throttling events, but the consumed write capacity is only at around 70% of what is allocated. Strangely, none of the GSIs report any throttling activity at all.

I've considered that there might be a hot key problem, but the partition key for this table is a UUID (stored as a string) and most of the traffic on this table is adding new entries, with only a small portion of the traffic updating existing data.

What else can I look at to try troubleshooting this issue?

pian0
  • 837
  • 1
  • 8
  • 19
  • 1
    I'd assume that you are hitting a single partition too frequently per https://aws.amazon.com/premiumsupport/knowledge-center/throttled-ddb/. Standard suggestions for dealing with this are in that same document. – jarmod Nov 15 '17 at 14:30
  • @jarmod It's a pretty small table, ~9 GB of data and reads and writes are less than 10. I wouldn't expect more than one partition at this size. – pian0 Nov 15 '17 at 15:10
  • Can you share RCU and WCU? And have you enabled Auto Scaling for write? – jarmod Nov 15 '17 at 15:38
  • @jarmod Table and 3 GSIs: RCU = 5, WCU = 6. Auto scaling is on for writes, 70% util, 3 min, 100 max. The behavior I see is write throttles start occuring on the table (not the indexes) as we approach 80% capacity. We see up to 2500 write throttle events per minute. – pian0 Nov 15 '17 at 16:53
  • 6 WCU represents capacity for 6 writes per second of up to 1KB per write, or 360x 1KB writes per minute. Also see https://stackoverflow.com/questions/16980678/when-does-dynamodb-throttle-request. – jarmod Nov 16 '17 at 01:41
  • 1
    You mention that it's a "pretty small table" and "wouldn't expect more than one partition" - but DDB will automatically partition behind the scenes based on its own internal calculations based on combination of table size, WCU, and RCU - and if it ever partitions, it won't "unpartition" it later. Based on your symptoms and description, that's most likely what's happened. – Krease Nov 16 '17 at 07:49
  • 1
    I agree with Krease. Maybe you temporarily increased the WCU during the initial data import (causing dynamo db to create extra partitions)? DDB will not undo this – xpa1492 Nov 17 '17 at 06:58

0 Answers0