0

I am new to AWS lambda and Kinesis. Please help with the following question

I have a kinesis stream as a source to lambda and the target is again kinesis. I have following queries. The system doesnt want to lose a record.

if any of the records fails the processing in lambda, How it again pull into the lambda? How it keep the unprocessed records ? How kinesis track the offset to process the next record?

Please update.

VinuBibin
  • 679
  • 8
  • 19

1 Answers1

1

From the AWS Lambda docs about using Lambda with Kinesis:

If your function returns an error, Lambda retries the batch until processing succeeds or the data expires. Until the issue is resolved, no data in the shard is processed. To avoid stalled shards and potential data loss, make sure to handle and record processing errors in your code.

In this context, also consider the Retention Period of Kinesis:

The retention period is the length of time that data records are accessible after they are added to the stream. A stream’s retention period is set to a default of 24 hours after creation. You can increase the retention period up to 168 hours (7 days)

As mentioned in the first quote, AWS will drop the event after the retention period is due. This means for you:

a) Take care that your Lambda function handles errors correctly.

b) If it's important to keep all records, also store them in a persistent storage, e.g. DynamoDB.

In addition to that, you should read about duplicate Lambda executions as well. There is a great blog post available explaining how you can achieve an idempotent implementation. And read here on another StackOverflow question & answer.

s.hesse
  • 1,900
  • 10
  • 13
  • How the lambda track this record is failed and should be processed it again ? Is there any built-in circuit breaker kind of to stop execution if its failing? Sorry to bombard with questions, am wondering how its internally managing – VinuBibin Sep 02 '19 at 15:12
  • I don't have any detailed insights, but it sounds like Lambda itself is not tracking if a record is failing, but instead Kinesis is doing that because they expect a successful response by the Lambda function. I never tried it with Kinesis, but for AWS Lambda in general there is an option to use a dead letter queue: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-lambda-function-deadletterconfig.html or https://aws.amazon.com/blogs/compute/robust-serverless-application-design-with-aws-lambda-dlq/ Not sure if that helps in your case. – s.hesse Sep 03 '19 at 17:15