Stack Overflow
Stack Overflow

Questions tagged [ksqldb]

ksqlDB: an event streaming database for Apache Kafka.

ksqlDB is a source-available event streaming database for Apache Kafka® developed by Confluent. It is distributed, scalable, reliable, and real-time. ksqlDB combines the power of real-time stream processing with the approachable feel of a relational database through a familiar, lightweight SQL syntax.

You can download ksqlDB at https://github.com/confluentinc/ksql

There is a quickstart at: https://ksqldb.io/quickstart.html

926 questions
7
votes
1 answer

Creating a KSQL Stream: How to extract value from complex json

I am trying to create a stream in Apache/KAFKA KSQL The topic contains (somewhat complex JSON) { "agreement_id": "dd8afdbe-59cf-4272-b640-b14a24d8234c", "created_at": "2018-02-17 16:00:00.000Z", "id": "6db276a8-2efe-4495-9908-4d3fc4cc16fa", …
Tobias Eriksson
  • 373
  • 4
  • 12
7
votes
1 answer

Read data from KSQL tables

maybe this is a beginner question but what is the recommended way to read data produced in KSQL? Let's assume I do some stream processing and write the data to a KSQL table. Now I want to access this data via a Spring application (e.g. fan-out some…
DerM
  • 1,497
  • 1
  • 14
  • 27
7
votes
3 answers

Counting all entries with KSQL

Is it possible to use KSQL to not only count entries of a specific column via GROUP BY but instead get an aggregate over all the entries that stream through the application? I'm searching for something like this: | Count all | Count id1 | count id2…
Tim.G.
  • 299
  • 4
  • 16
6
votes
1 answer

KSQL in AWS MSK

Does anyone have any idea if AWS MSK (Managed Streaming for Kafka) supports KSQL? I have seen many videos and documentations for KSQL on confluent platform but not for AWS MSK. Please let me know if it supports and also let me know if there is any…
Bala
  • 99
  • 2
  • 6
6
votes
2 answers

Best way to join two (or more) kafka topics in KSQL emiting changes from all topics?

We have a "microservices" platform and we are using debezium for change data capture from databases on these platforms which is working nicely. Now, we'd like to make it easy for us to join these topics and stream the results into a new topic which…
Phillip Fleischer
  • 1,023
  • 12
  • 14
6
votes
1 answer

KSQL Windowed Aggregation Stream

I am trying to group events by one of its properties and over time using the KSQL Windowed Aggregation, specifically the Session Window. I have a STREAM made from a kafka topic with the TIMESTAMP property well specified. When I try to create a…
Filippo Vitale
  • 7,597
  • 3
  • 58
  • 64
6
votes
1 answer

Is it possible to create ksql table from ksql stream?

I'm a new bee to ksql. I'm just playing with read kafka topics to streams and it works great. Also, trying to create a table from kafka topic and failed. Realized that I need to have a key set in kafka topic which is considered as primary key in…
srikanth
  • 958
  • 16
  • 37
6
votes
3 answers

How to join multiple Kafka topics?

So I have... 1st topic that has general application logs (log4j). Stores things like HTTP API requests/responses and warnings, exceptions etc... There can be multiple logs associated to one logical business request. (These logs happen within…
user432024
  • 4,392
  • 8
  • 49
  • 85
5
votes
2 answers

Field does not exist on transformations to extract key with Debezium

I am trying to create a Debezium MySQL connector with a transformation to extract the key. Before key transformations: create source connector mysql with( "connector.class" = 'io.debezium.connector.mysql.MySqlConnector', "database.hostname"…
4it med
  • 181
  • 1
  • 7
5
votes
2 answers

Is there a way to consume a Kafka Ksql Push query from .NET

I'm processing a large volume of Kafka messages in .NET with a Kafka consumer at the moment. Step 1 in my processing is to Parse the JSON and to discard many of the messages based on the value of a specific field in the JSON. I'd like to not process…
Workerbee
  • 53
  • 1
  • 3
5
votes
2 answers

Kafka time difference last two records, KSQL or other?

So I'm evaluating Kafka. In our use case would have to create new topics containing "time elapsed" from one event to the next, essentially since a sensor will report as "on" or "off" into Kafka. So having the timestamp, sensorname and state, create…
Thinus Marloth
  • 51
  • 3
5
votes
1 answer

How can I create a KSQL table from a topic using a composite key?

Say I have a topic with temperature forecast data, as…
Mike Williams
  • 316
  • 1
  • 6
5
votes
1 answer

How to create KSQL Stream with large number of JSON fields from topic in kafka?

I am passing a long JSON String to kafka topic eg: { "glossary": { "title": "example glossary", "GlossDiv": { "title": "S", "GlossList": { "GlossEntry": { "ID": "SGML", …
Mehul Gupta
  • 440
  • 5
  • 23
5
votes
1 answer

Confluent 4.1.0 ->KSQL : STREAM-TABLE join -> table data null

STEP 1: Run the producer to create sample data ./bin/kafka-avro-console-producer \ --broker-list localhost:9092 --topic stream-test-topic \ --property schema.registry.url=http://localhost:8081 \ --property…
Zamir Arif
  • 341
  • 2
  • 13
4
votes
0 answers

ksqldb REST API push query - reconnect to last consumed offset + 1

Is it possible to reconnect to the same push query after connection was lost? There is a queryId for terminating a query. Is there a similar mechanism which could be used for reconnections via ksqldb REST API? In case that the subscription stopped…
Kubus
  • 677
  • 6
  • 18
1
2 3
…
61 62