Stack Overflow
Stack Overflow

Questions tagged [amazon-kinesis-kpl]

52 questions
14
votes
1 answer

Amazon Kinesis KPL vs AWS SDK pros and cons

The scenario is I would be writing large volumes of data ( terabytes per day) to kinesis stream.I want to know which is a better way to achieve high write throughput. I am considering the below two options for producer clients. Option 1: using…
yin yang
  • 209
  • 5
  • 14
6
votes
0 answers

How to fetch the audio file from Kinesis Video stream created from Amazon connect in proper format?

I am trying to setup a VOICE MAIL feature for the callcenter in Amazon connect.I have added the Start Media streaming block to the flow.I have also added a Lambda function which is triggered by the Kinesis stream.My idea is NOT to do a Live…
James_RajKumar
  • 201
  • 3
  • 12
5
votes
1 answer

spark kinesis failing on cloudera with java.lang.AbstractMethodError

below is my POM file. I am writing a spark streaming with aws kinesis org.apache.spark spark-streaming_2.10 1.6.0 …
Karn_way
  • 1,005
  • 3
  • 19
  • 42
4
votes
3 answers

Kinesis vs KPL vs KCL

This is somewhat of a shallow-level question. However, I perplexed by this trio of services. I understand that KPL produces fast data and KCL consumes fast data produced by Kinesis. However, what I fail to understand is the if KPL and KCL make up…
Mamun
  • 2,322
  • 4
  • 27
  • 41
4
votes
4 answers

How to convert a Data Class to ByteBuffer in Kotlin?

I am trying to use Kinesis, which expects data in byte buffer format. All the examples I have seen so far are in Java and pass simple strings. Can anybody give an idea of how to convert a kotlin data class to bytebuffer? e.g. data class abc ( …
N. S. Mehra
  • 45
  • 1
  • 1
  • 5
4
votes
2 answers

Call REST API for Amazon Kinesis with Setting up API Gateway

I am trying to send a HTTP Post Request to put a record into Amazon Kinesis Stream. There are several ways (Kinesis Client, KPL, setting up AWS Gateway as Kinesis Proxy). I saw this document about Kinesis PutRecord…
needhelp
  • 87
  • 1
  • 7
3
votes
2 answers

Loading multiple records to Kinesis using PutRecords - how to re-send only failed records in case of failure?

I’m using Lambda to load data records into Kinesis and often want to add up to 500K records, I am batching these into chunks of 500 and using Boto's put_records method to send them to Kinesis. I sometimes see failures due to exceeding the allowed…
Craig Watkinson
  • 903
  • 10
  • 20
3
votes
2 answers

How does AWS Kinesis throttle write throughput?

AWS Kinesis has a fairly low write throughput of 1000 writes/sec and 1MB/writes-sec. How does Kinesis enforce this limit? If I were to try to do 1500 writes in a second, would the extra 500 writes be placed into some sort of queue or would they…
Justin Borromeo
  • 1,201
  • 3
  • 13
  • 26
3
votes
2 answers

Disable INFO logs in aws kcl - Kinesis

I'm running on Ubuntu 16.04 & Java 8. kcl generate thousands of INFO log lines. Does any one know how to enable only ERROR and WARN logs? *I have also the same question for kpl. I don't have a logfile.
Stav Alfi
  • 13,139
  • 23
  • 99
  • 171
3
votes
2 answers

Kinesis Firehose Putrecordbatch example

I am looking for an example to send multiple records to kinesis streams using putrecordbatch. I am currently using putrecord command in the following way to send records to kinesis streams. aws firehose put-record --delivery-stream-name…
Srivignesh KN
  • 452
  • 8
  • 22
3
votes
1 answer

Reading from Kinesis stream with more than 1 shard in Spark Streaming

Getting following error: com.amazonaws.services.kinesis.clientlibrary.exceptions.ShutdownException: Can't update checkpoint - instance doesn't hold the lease for this shard What could be the cause of this?
Ronald Ángel Marín
  • 31
  • 3
2
votes
1 answer

How to disable CloudWatch metrics for KPL/KCL with Spring Cloud Stream

I am using the Spring Cloud Stream Binder for Kinesis with KPL/KCL enabled. We would like to disable Cloudwatch metrics without having to manage the configuration of KPL and KCL ourselves (completely overriding the beans). We would like to use the…
Amrit Baveja
  • 528
  • 7
  • 13
2
votes
0 answers

Does KPL support proxy configurations?

I have Spring beans configured in a microservice as follows that I'm using to connect to Kinesis: @Bean // ClientConfiguration has proxy protocol, proxy host, and proxy port set public AmazonKinesisAsync amazonKinesisAsync(ClientConfiguration…
Keith Bennett
  • 733
  • 11
  • 25
2
votes
0 answers

What is the KPL protocol?

https://docs.aws.amazon.com/streams/latest/dev/kinesis-kpl-integration.html The Kinesis Producer Library (KPL) runs in a separate process, and communicates with your parent user process using IPC. This architecture is sometimes called a…
Paul Draper
  • 78,542
  • 46
  • 206
  • 285
2
votes
0 answers

Multiple sources for kinesis firehose

Basically, I have servers writing data into kinesis streams in multiple regions. I would want to read from those streams and dump the data into a single S3 bucket, preferably consolidated one. I was hoping to make kinesis firehose read from…
vinod ep
  • 51
  • 1
  • 7
1
2 3 4