Stack Overflow
Stack Overflow

Questions tagged [event-stream-processing]

43 questions
7
votes
2 answers

Stream processing architecture

I am in the process of designing a system where there's a main stream of objects and there are multiple workers which produces some result from that object. Finally, there is some special/unique worker (sort of a "sink", in terms of graph theory)…
IsaacLevon
  • 2,260
  • 4
  • 41
  • 83
6
votes
1 answer

Handling event streams in haskell

I want to process stream of events received via the MQTT. Library which I'm using uses a callback to provide the results. Processing I'm doing depends on the previous state not only the latest event. Also in the future events might be gathered from…
majkrzak
  • 1,332
  • 3
  • 14
  • 30
4
votes
2 answers

Cost of an unused Kafka topic/partition

In designing a streaming processing pipeline what cost might be incurred if I were to have many topics which would have at least one partition but potentially no data going into it? As an example, with one consumer and I could choose to have one…
cwadding
  • 870
  • 11
  • 16
4
votes
1 answer

Is RethinkDB a good fit for a generic Real-time aggregation platform?

I need your help to verify if RethinkDB fits my use case. Use case My team is building a generic Real-time aggregation platform which needs to: join data from a lot of Kafka topics Joins need to be done on raw data Topics have the same key Data in…
davorp
  • 4,156
  • 3
  • 26
  • 34
3
votes
2 answers

Apache Flink - exception handling in "keyBy"

It may happen that data that enters Flink job triggers exception either due to bug in code or lack of validation. My goal is to provide consistent way of exception handling that our team could use within Flink jobs that won't cause any downtime in…
Mariusz B
  • 31
  • 3
3
votes
3 answers

nginx buffering flask event-stream in Docker image

I have a REST API backend with python/flask and want to stream the response in an event stream. Everything is running inside a docker container with nginx/uwsgi (https://hub.docker.com/r/tiangolo/uwsgi-nginx-flask/). The API works fine until it…
SumNeuron
  • 4,850
  • 5
  • 39
  • 107
2
votes
4 answers

Stream processing alternatives

We have a few thousand IoT devices that send us their temperature every second. The input source can be MQTT or JSON (or a queue if needed). Our goal is to near continuously process data for each of these devices and calculate the average for the…
NorwegianClassic
  • 935
  • 8
  • 26
2
votes
1 answer

AggregateFunction getResult() not been called after event aggregation

Trying to implement a Flink job for reading Kafka stream and aggregating the session, for some reason getResult() is not being called. I see createAccumulator() and add() were called, I'm expecting getResult() also be called so that I can sink…
Sparkle8
  • 225
  • 3
  • 13
2
votes
2 answers

How to update a stream with the response from another stream where the sink type is "http-response"

Am trying to enrich my input stream with an additional attribute which gets populated via "http-response" response sink. I have tried using the "join" with window attribute and with "every" keyword to merge two streams and inserting the resulting…
Vigneshwaran
  • 45
  • 5
2
votes
2 answers

Generate "fake" stream data. Kafka - Flink

I am trying to generate stream data, to simulate a situation where I receive two values, Integer type, in a different time range, with timestamps, and Kafka as connector. I am using Flink environment as a consumer, but I don't know which is the…
Salvador Vigo
  • 397
  • 4
  • 16
1
vote
2 answers

WSO2 Stream Processor (SP) Error "could not initialize carbon deployment engine"

I have an issue with my web IDE. The WSO2 SP or siddhi-tooling-5.1.2 starts up as you would expect and I can open the website. However, in the console upon startup it gives this error. This is an issue as I am unable to interact with the web IDE…
Casper Andersen
  • 21
  • 4
1
vote
1 answer

Sink for user activity data stream to build Online ML model

I am writing a consumer that consumes (user activity data, (activityid, userid, timestamp, cta, duration) from Google Pub/Sub and I want to create a sink for this such that I can train my ML model in online fashion. Since this sink is the source…
amor.fati95
  • 119
  • 1
  • 11
1
vote
1 answer

How to produce messages with consecutive numbers with Kafka?

Context: invoicing system, the sent invoices must have consecutive numbers. Each invoice has a unique invoice number, for sake of simplicity let's say they are I1, I2, I3, and so on. So, the first invoice in the system has the number I1, and it gets…
Eduard Kukuy
  • 11
  • 3
1
vote
1 answer

On-Demand Query with siddhi fails

I'm using the wso2 stream integrator in order to run my siddhi application. The operating environment is windows 10 pro. I'd like to ask you for your help about way of using Siddhi's REST API. Open a command prompt as the administrator and go to…
Light
  • 17
  • 5
1
vote
1 answer

Merge attributes from two streams in siddhi

My intention is to merge attributes from two streams in siddhi. I'm using the "join" with window attribute to merge two streams in my siddhi query and inputting the result of the join to another stream to enrich it. The window attributes…
Vigneshwaran
  • 45
  • 5
1
2 3