How to consume/read messages from KAFKA topic with filter condition

Question

I have a topic with different messages. But I need to consume only certain messages irrespective of the order in which message published. I need to apply data filter condition on specific field/attribute of the message. like file-name element of the JSON message is "XXXX.txt". Simply say consume message with where condition like traditional DB.

Answer 1

.I need to use C# client

Then your only option would be to subscribe and poll like normal, adding an if statement for your condition

Borrowing snippet from the Confluent consumer example

 // TODO: build consumer that reads string messages 
consumer.Subscribe(topics);
... 
while (true) {
    var consumeResult = consumer.Consume(cancellationToken);
    string value = consumeResult.Value;

    // TODO: parse string value to JSON object using your favorite JSON library 
    // Add your condition 
    if (jsonObj["file-name"].Equals("XXXX.txt"))
    {
        Console.WriteLine($"Received message at {consumeResult.TopicPartitionOffset}: {value}");
    }

Regarding the JSON parser, you can either make a Dictionary or a strongly typed class, but this example assumes a dictionary. How can I deserialize JSON to a simple Dictionary<string,string> in ASP.NET?

irrespective of the order in which message published.

As with any consumer, order is guaranteed within a partition