0

This is a project that is being developed using AWS.

I have scheduled my lambda function using the cron expression in CloudWatch. The function will upload items to DynamoDB daily.

Some items are not uploaded to Dynamodb despite having a unique primary key. Sometimes consecutive items are skipped, sometimes items with slightly similar primary keys are skipped. Usually, the number of items skipped is below 20.

It fully works when I run the lambda function manually again. Would like to know the reason behind this and possibly the solution.Thanks!

  • 1
    when you say lambda skips some items, did you tried to check its logs, under cloudwatch? – Jatin Mehrotra Sep 11 '21 at 04:15
  • @JatinMehrotra I did a try-except for the batchwriteitems response, and only displays it when it reaches the except section. All the responses in the except section has 0 unprocessed items. –  Sep 11 '21 at 04:33
  • 1
    there must be some error try logging that error in except block, and see it in cloudwatch https://stackoverflow.com/q/33068055/13126651 – Jatin Mehrotra Sep 11 '21 at 05:03

1 Answers1

0

The BatchWriteItem documentation explains that if the database encounters problems - most notably if your request rate exceeds your provisioned capacity - it is likely that a BatchWriteItem will only successfully write some of the items in the batch. The list of the items not written will be returned in a UnprocessedItems attribute in the response, and you are expected to try again to write those same unprocessed items:

If any requested operations fail because the table's provisioned throughput is exceeded or an internal processing failure occurs, the failed operations are returned in the UnprocessedItems response parameter. You can investigate and optionally resend the requests. Typically, you would call BatchWriteItem in a loop. Each iteration would check for unprocessed items and submit a new BatchWriteItem request with those unprocessed items until all items have been processed.

If none of the items can be processed due to insufficient provisioned throughput on all of the tables in the request, then BatchWriteItem returns a ProvisionedThroughputExceededException.

If DynamoDB returns any unprocessed items, you should retry the batch operation on those items. However, we strongly recommend that you use an exponential backoff algorithm. If you retry the batch operation immediately, the underlying read or write requests can still fail due to throttling on the individual tables. If you delay the batch operation using exponential backoff, the individual requests in the batch are much more likely to succeed.

Nadav Har'El
  • 11,785
  • 1
  • 24
  • 45