Stack Overflow
Stack Overflow

Questions tagged [amazon-kinesis-analytics]

Amazon Kinesis Data Analytics is the way to analyze streaming data, gain actionable insights, and respond in real time. SQL users can query streaming data or build entire streaming applications using templates and an interactive SQL editor. Java developers can build streaming applications using open source Java libraries and AWS integrations to transform and analyze data in real-time.

133 questions
4
votes
1 answer

Watermark strategy on Flink Kinesis Consumer

I am investigating using Flink with a Kinesis stream as a source. I would like to use Event Time watermarking. Planning on running this on AWS managed Flink (Kinesis Analytics) platform. Looking at the AWS documentation and indeed Flink…
Bombington
  • 83
  • 4
4
votes
1 answer

How to deploy an AWS Kinesis Data Analytics App without downtime

We currently have an AWS Kinesis Data Analytics app that requires a .jar file to run. We have automated the deployment for our .jar file that resides in an S3 bucket. Our issue is, whenever the .jar file is updated we are forced to restart the…
Marc Asmar
  • 1,527
  • 1
  • 10
  • 22
4
votes
1 answer

Kinesis Data Analytics Flink: Continually Increasing Checkpoint Size

I am running a Flink application using the AWS Kinesis Data Analytics (KDA) service. My KDA Flink application last checkpoint size appears to be growing steadily over time. The sudden drops in checkpoint size you can see in the attached graph…
ChrisATX
  • 109
  • 1
  • 11
4
votes
1 answer

How to stop high load from leading to cascading Flink checkpoint failures

A couple of points i'll volunteer up front: I'm new to Flink (working with it for about a month now) I'm using Kinesis Analytics (AWS hosted Flink solution). By all accounts this doesn't really limit the versatility of Flink or the options for…
Wendell Blatt
  • 177
  • 1
  • 1
  • 10
3
votes
1 answer

How to provide KafkaSource SSL files to Flink worker nodes

I am creating a Kafka-based Flink streaming application, and am trying to create an associated KafkaSource connector in order to read Kafka data. For example: final KafkaSource source = KafkaSource.builder() // standard source…
pika
  • 93
  • 1
  • 6
3
votes
0 answers

AWS Kinesis Data Analytics: PyFlink with nested JSON data

With Kinesis analytics SQL application, we have the option to configure the input Schema. So that we can map the input data to flat structure. So I'm looking for the similar configuration with Kinesis Data Analytics Apache Flink…
Dhinsha Mahesh
  • 41
  • 4
3
votes
2 answers

Deploy a Python Flink application on AWS Kinesis

I am trying to deploy a Python Flink application on AWS Kinesis Data Analytics. I followed the official documentation on https://docs.aws.amazon.com/kinesisanalytics/latest/java/how-python-creating.html I want to create a source table using the…
Adrian Chang Alcover
  • 31
  • 3
3
votes
1 answer

Passing elements back to the input stream, after processing, in Flink?

Scenario: I have a stream of events coming from the sensor. The Event could be of T-type or J-Type. T-type events have event occurred timestamp. J-type events have a start and end timestamp. Based on the start and end timestamp of J-Type event,…
Arjun Sunil Kumar
  • 1,781
  • 3
  • 28
  • 46
3
votes
3 answers

Amazon Kinesis Data Analytics for Java Applications: Avro issue in deserialization incoming messages

I tried to deploy my Flink application into AWS Kinesis Data Analytics. This application use Apache Avro for deserialization/serialization incoming messages. My application works fine on my local machine, but when I deploy it to AWS, I have…
Pavel Moriakov
  • 61
  • 5
3
votes
0 answers

AWS Kinesis Analytics aggregate tumbling time windows results

In AWS Kinesis Analytics, how to aggregate the results between tumbling time windows ? Let's say there are 10 seconds tumbling time windows, like in the given example from AWS website: CREATE OR REPLACE STREAM "DESTINATION_SQL_STREAM"…
kord
  • 979
  • 14
  • 24
2
votes
0 answers

Get live video from Amazon KVS

I am trying to get live video stream from the Amazon KVS to show in a dashboard board that I am building using React. I am very new to this (Amazon KVS)ecosystem and have no idea about how things work hence asking you good folks here. I tried…
Jack Sparrow
  • 21
  • 1
2
votes
0 answers

Kinesis Data Analytics - Flink state serializer incompatible after recovering from Snapshot

We have our Flink application(version 1.13.2) deployed on AWS KDA. The strategy is that we do not want the application to stop at all, so we always recover the application from a snapshot when updating the jar with new changes. Recently, we found a…
Charles fu
  • 21
  • 2
2
votes
1 answer

Run Athena every 15 minutes vs Kinesis Data Analytics

I am going to be using Athena for report generation on data available in S3. A lot of it is time series data coming from IoT devices. Users can request reports over years and years' worth of data but will mostly be weekly, monthly or annual. I am…
systemdebt
  • 4,589
  • 10
  • 55
  • 116
2
votes
0 answers

"Could not execute application" for streaming application deployed from Kinesis Analytics Studio

I have a Kinesis Analytics Studio notebook that is working well, and I'm trying to deploy the solution as an application by following the steps here (the steps for console), but I'm running into a few problems. The notebook contains a python udf…
Siyaj K'ak'
  • 21
  • 2
2
votes
1 answer

How to inject delay between the window and sink operator?

Context - Application We have an Apache Flink application which processes events The application uses event time characteristics The application shards (keyBy) events based on the sessionId field The application has windowing with 1 minute tumbling…
Peter Csala
  • 17,736
  • 16
  • 35
  • 75
1
2 3
…
8 9