Stack Overflow
Stack Overflow

Questions tagged [amazon-kcl]

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

The Amazon Kinesis Client Library (KCL) helps you consume and process data from an Amazon Kinesis stream. This type of application is also referred to as a consumer.

Source: https://docs.aws.amazon.com/streams/latest/dev/developing-consumers-with-kcl.html

99 questions
47
votes
4 answers

TRIM_HORIZON vs LATEST

I can't find in the formal documentation of AWS Kinesis any explicit reference between TRIM_HORIZON and the checkpoint, and also any reference between LATEST and the checkpoint. Can you confirm my theory: TRIM_HORIZON - In case the application-name…
Ida Amit
  • 1,411
  • 2
  • 13
  • 27
20
votes
3 answers

Amazon KCL Checkpoints and Trim Horizon

How are checkpoints and trimming related in AWS KCL library? The documentation page Handling Startup, Shutdown, and Throttling says: By default, the KCL begins reading records from the tip of the stream;, which is the most recently added record.…
Edmondo
  • 19,559
  • 13
  • 62
  • 115
7
votes
1 answer

How to handle reprocessing scenarios in AWS Kinesis?

I am exploring AWS Kinesis for a data processing requirement that replaces old batch ETL processing with a stream based approach. One of the key requirements for this project is the ability to reprocess data in cases when A bug is discovered and…
Rahul
  • 12,886
  • 13
  • 57
  • 62
7
votes
0 answers

Kinesis Shard GetRecords.IteratorAgeMilliseconds reached maximum 86.4M (1 day) and does not decrease even though consuming

I am consuming a Kinesis stream with Spark Streaming 2.2.0 and using spark-streaming-kinesis-asl_2.11. Kinesis Stream has 150 shards and I am monitoring GetRecords.IteratorAgeMilliseconds CloudWatch metric to see whether consumer is keeping up with…
Grega Kešpret
  • 11,827
  • 6
  • 39
  • 44
7
votes
1 answer

Kinesis client library record processor failure

According to AWS docs: The worker invokes record processor methods using Java ExecutorService tasks. If a task fails, the worker retains control of the shard that the record processor was processing. The worker starts a new record processor task to…
Aparna P L
  • 147
  • 1
  • 2
  • 7
7
votes
1 answer

How to use ExplicitHashKey for round robin stream assignment in AWS Kinesis

I am trying to pump lots of data through Amazon Kinesis (order 10,000 points per second). In order to maximize records per second through my shards, I'd like to round robin my requests over the shards (my application logic doesn't care what shard…
deadcode
  • 2,226
  • 1
  • 20
  • 29
5
votes
1 answer

Use Kinesis Client Library v2.x with DynamoDB Stream Kinesis Adapter

I'm building a Java application to consume events from a DynamoDB Stream using the KCL library. The example on AWS docs uses this library: com.amazonaws:dynamodb-streams-kinesis-adapter:1.4.0 Which depends…
dipanda
  • 760
  • 1
  • 11
  • 24
5
votes
1 answer

AWS Kinesis Stream Checkpointing

I have an application that's capable of handling duplicate Kinesis stream records. We're considering the approaches we could take in terms of handling failures. And the following approach was brought up: If an exception is caught during…
ddolce
  • 739
  • 2
  • 10
  • 30
5
votes
0 answers

KCL consumer not processing records

I've gone through several issues and SO questions and haven't been able to get the sample app to work. Here's how to reproduce this; git clone https://github.com/awslabs/amazon-kinesis-client-python.git cd amazon-kinesis-client-python # Start…
Jonathan
  • 10,792
  • 5
  • 65
  • 85
5
votes
1 answer

How can I measure the propagation latency of DynamoDB Streams?

I'm using DynamoDB Streams + Kinesis Client Library (KCL). How can I measure latency between when an event was created in a stream and when it was processed on KCL side? As I know, KCL's MillisBehindLatest metric is specific to Kinesis Streams(not…
Ivan Arkhipov
  • 51
  • 3
5
votes
2 answers

Dump Kinesis Client Library (KCL) logs to file

Am using Kinesis Client Library (KCL) to subscribe to kinesis stream. All the KCL logs are printed on console. I need to dump all the logs to file. I tried adding log4j.properties and common-logging.properties files in src folder, but not able to…
Sunil Gulabani
  • 878
  • 1
  • 8
  • 21
4
votes
1 answer

Dynamodb stream kinesis - Incomplete hash range found between

I have kinesis stream from DynamoDB. And I am processing it through aws KCL sdk v1.14.0. I see below occasional errors in the logs. I also observe that startingHashKey is always 1 and endingHashKey is 0 for all shards in the dynamodb lease table.…
jzqa
  • 843
  • 10
  • 18
4
votes
1 answer

Load balancing and scaling in Kinesis Client library (KCL) based application

I am using amazon-kinesis-connectors to build a kinesis client side application. I am figuring out few things about it. How KCL ensures load balancing and scaling. for example i have a stream with one shard and a lot of records/events are put for 15…
Awadesh
  • 3,530
  • 2
  • 20
  • 32
4
votes
2 answers

kinesis getting data from multiple shards

I am trying to build a simple application that reads data from AWS Kinesis. I have managed to read data using a single shard but I want to get data from 4 different shards. Problem is, I have a while loop which iterates as long as the shard is…
emrahozkan
  • 193
  • 1
  • 3
  • 15
3
votes
1 answer

Does Testcontainers/LocalStack work with DynamoDb Streams KCL 1.x?

Problem Statement I have written a program which utilizes DynamoDb Streams to get notified when an update occurs, this code works fine when using AWS Services but doesn't seem to work utilizing Testcontainers/Localstack for my integration…
Jim M.
  • 909
  • 1
  • 12
  • 29
1
2 3 4 5 6 7