2

We are experiencing a ProvisionedThroughputExceededException upon checkpointing many events together.

The exception stacktrace is the following:

    com.amazonaws.services.kinesis.model.ProvisionedThroughputExceededException: Rate exceeded for shard shardId-000000000000 in stream mystream under account accountid. (Service: AmazonKinesis; Status Code: 400; Error Code: ProvisionedThroughputExceededException; Request ID: ea36760b-9db3-0acc-bbe9-87939e3270aa)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1529)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1167)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:948)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:635)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:618)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$300(AmazonHttpClient.java:586)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:573)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:445)
at com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:1645)
at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:1621)
at com.amazonaws.services.kinesis.AmazonKinesisClient.getShardIterator(AmazonKinesisClient.java:909)
at com.amazonaws.services.kinesis.clientlibrary.proxies.KinesisProxy.getIterator(KinesisProxy.java:291)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.SequenceNumberValidator.validateSequenceNumber(SequenceNumberValidator.java:79)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.RecordProcessorCheckpointer.checkpoint(RecordProcessorCheckpointer.java:120)
at com.amazonaws.services.kinesis.clientlibrary.lib.worker.RecordProcessorCheckpointer.checkpoint(RecordProcessorCheckpointer.java:90)

As you can see here, the exception is raised at

RecordProcessorCheckpointer.java:90

inside the KCL library. What does checkpointing has to do with exceeding the throughput?

Edmondo
  • 19,559
  • 13
  • 62
  • 115

1 Answers1

2

Kinesis is rate-limited,

PutRecord requests can only process up to the limit of the provisioned throughput on the involved shard. exceeding this will throw ProvisionedThroughputExceededException

Obvious solution would be splitting stream's shard into two and divide the hash key space evenly. It might look unnecessary if your metrics are within the limits of a single shard but lets say if you use your limit of 1000 transactions/sec write capacity in first 500ms your activity for that shard will be throttled for the remaining half so there is no way you can avoid throttling with a single shard.

You can configure automatic retries after short delays for your throttled requests. check your SDK's documentation if there is any examples of this.

Ulug Toprak
  • 1,172
  • 1
  • 10
  • 21
  • Checkpointing does not throw this exception, you are only filling an extra column for each row with the associated sequence number by checkpointing... `ProvisionedThroughputExceededException` is only related to exceeding throughput limits. – Ulug Toprak Jul 12 '17 at 11:12
  • an extra row where ? are we actually adding s new record to kinesis when checkpointing? – Edmondo Jul 12 '17 at 11:23
  • Its hard to help without seeing your code, here is a sample record processor code written in python it might be some help to you. [kclpy](https://github.com/awslabs/amazon-kinesis-client-python/blob/master/samples/sample_kclpy_app.py) – Ulug Toprak Jul 12 '17 at 12:37
  • The exception is raised on IRecordProcessorCheckpointer.checkpoint invocation – Edmondo Jul 12 '17 at 13:25
  • Sorry man this turned into a guessing game, if you don't provide your code or error logs its not gonna be possible for me to help you. – Ulug Toprak Jul 12 '17 at 13:54
  • It seems I have the same issue. When it hits ProvisionThroughputExceededException, the KCL recordProcessor will stop processing until the next restart. The checkpoint is call validate as the stacktrace below: `com.amazonaws.services.kinesis.clientlibrary.exceptions.ThrottlingException: Exceeded throughput while getting an iterator for shard shardId-000000000005 at com.amazonaws.services.kinesis.clientlibrary.lib.worker.SequenceNumberValidator.validateSequenceNumber(SequenceNumberValidator.java:89) ~[amazon-kinesis-client-1.9.0.jar:na]` – ethan May 31 '18 at 23:34