Stack Overflow
Stack Overflow

Questions tagged [flink-cep]

FlinkCEP - complex event processing for Apache Flink

368 questions
11
votes
1 answer

Flink: How to handle external app configuration changes in flink

My requirement is to stream millions of records in a day and it has huge dependency on external configuration parameters. For example, a user can go and change the required setting anytime in the web application and after the change is made, the…
Neoster
  • 195
  • 2
  • 11
8
votes
11 answers

No ExecutorFactory found to execute the application in Flink 1.11.1

first of all I have read this post about the same issue and tried to follow the same solution that works for him (create a new quickstart with mvn and migrate the code there) and is not working eighter when out-of-the-box of IntelliJ. Here is my…
Alter
  • 903
  • 1
  • 11
  • 27
7
votes
4 answers

Best approach to construct a real-time rule engine for our streaming events

We are at the beginning of building an IoT cloud platform project. There are certain well known portions to achieve complete IoT platform solution. One of them is real-time rule processing/engine system which is needed to understand that streaming…
ccobanoglu
  • 192
  • 3
  • 13
6
votes
3 answers

how to make sure that flink job has finished executing and then perform some tasks

I want to perform some tasks after flink job is completed,I am not having any issues when I run code in Intellij but there are isssues when I run Flink jar in a shell file. I am using below line to make sure that execution of flink program is…
Amarjit Dhillon
  • 2,718
  • 3
  • 23
  • 62
5
votes
2 answers

Apache Flink Resource Planning best practices

I'm looking for recommendations/best practices in determining required optimal resources for deploying a streaming job on Flink Cluster. Resources are No. of tasks slots per TaskManager Optimal Memory allocation for TaskManager Max Parallelism
ardhani
  • 303
  • 1
  • 11
5
votes
1 answer

Is there a way to determine total job parallelism or number of slots required to run a Flink job(before it is run)

Is there a way to determine the total number of task slots that will be required to run the job from either the execution plan or in some other way without having to actually start the job first. According to this doc:…
SherinThomas
  • 1,881
  • 4
  • 16
  • 20
5
votes
2 answers

How to debug serializable exception in Flink?

I've encountered several serializable exceptions, and I did some searching on Flink's internet and doc; there are some famous solutions like transient, extends Serializable etc. Each time the origin of exception is very clear, but in my case, i am…
Leyla Lee
  • 466
  • 5
  • 19
5
votes
1 answer

Flink exactly-once message processing

I've setup a Flink 1.2 standalone cluster with 2 JobManagers and 3 TaskManagers and I'm using JMeter to load-test it by producing Kafka messages / events which are then processed. The processing job runs on a TaskManager and it usually takes ~15K…
razvan
  • 559
  • 7
  • 23
5
votes
1 answer

How to configure Flink to use Hdfs for backend state and checkpoints

I have a setup with Flink v1.2, 3 JobManagers, 2 TaskManagers. I want to use hdfs for backend state and checkpoints and zookeeper storageDir state.backend: filesystem state.backend.fs.checkpointdir: hdfs:///[ip:port]/flink-checkpoints …
razvan
  • 559
  • 7
  • 23
5
votes
2 answers

Apache Flink CEP Pattern operation for NOT followedBy

I have a scenario where I have to change the state if a second event did not follow first event within x seconds. For e.g. user did not logout in 100 mins, consider him to be in invalid state. How can this be designed using the current pattern…
Sowmya V
  • 53
  • 5
4
votes
2 answers

Flink Unit Test over ProcessWindowFunction

How can I create a unit test for a Stateful Process Function. I have something like this: private static SingleOutputStreamOperator methodName(KeyedStream stream) { return…
Alter
  • 903
  • 1
  • 11
  • 27
4
votes
2 answers

Using Python user defined function in a Java Flink Job

Is there anyway to use a python user defined function within a Java Flink Job or anyway to communicate for example the result of a transformation done by flink with java with a python user defined function to apply some machine learning things: I…
Alter
  • 903
  • 1
  • 11
  • 27
4
votes
1 answer

Flink CEP Pattern does not match for first events after starting job and always matches previous events set

I want to match a CEP Pattern in Flink 1.4.0 Streaming with the following code: DataStream input = inputFromSocket.map(new IncomingMessageProcessor()).filter(new FilterEmptyAndInvalidEvents()); DataStream inputFiltered =…
sceee
  • 1,681
  • 19
  • 34
4
votes
1 answer

Dynamically throttle flink kafka sources

We're consuming multiple kafka topics but want to give precedence to some of them (~ Quality of Service). According to what I've found online, the consensus is to not throttle in operators but in the source, more specifically the deserializer…
Niklas Fasching
  • 1,326
  • 11
  • 15
4
votes
1 answer

Is it possible to generate watermarks per key in a KeyedStream (Apache Flink)?

I'm working on implementing a use case wherein different physical devices are sending events, and due to network/power issues, there can be a delay in receiving events at flink source. One of the operators within the flink job is the Pattern…
shailesh
  • 73
  • 4
1
2 3
…
24 25