31

I am investigating Kafka 9 as a hobby project and completed a few "Hello World" type examples.

I have got to thinking about Real World Kafka applications based on request response messaging in general and more specifically how to link a Kafka request message to its response message.

I was thinking along the lines of using a generated UUID as the request message key and employ this request UUID as the associated response message key. Much the same type of mechanism that WebSphere MQ has message correlation id.

My end 2 end process would be.

1). Kafka client generates a random UUID and sends a single Kafka request message. 2). The server would consume this request message extract & store the request UUID value 3). complete a Business Process using the message payload. 4). Respond with a response message that employs the stored UUID value from the request message as response message Key. 5). the Kafka client polls the response topic until it either timeouts or retrieves a message with the original request UUID value.

What I concerned about is that the Kafka Consumer polling will remove other clients messages from the response topic and increment the offsets making other clients fail.

Am I trying to apply Kafka in a use case it was never designed for?

Is it possible to implement request/response messaging in Kafka?

Jasper Siepkes
  • 1,412
  • 16
  • 25
Hector
  • 4,016
  • 21
  • 112
  • 211

4 Answers4

12

Even though Kafka provides convenience methods to persist the committed offsets for a given consumer group, you're not required to use that behavior and can write your own if you feel the need. Even so, the use of Kafka the way you've described it is a bit awkward for the use case as each client needs to repeatedly search the topic for a specific response. That's inefficient at best.

You could break the problem into two parts, continuing to use Kafka to deliver requests to and responses from your server. The only piece you'd need to add would be some sort of API layer that your clients talk to and which encapsulates the Kafka-specific logic from your clients. This layer would need a local DB (relational or NoSQL) that could store responses by uuid making it very fast and easy for the API to answer whether a response is available for a specific uuid.

Chris Gerken
  • 16,221
  • 6
  • 44
  • 59
  • 2
    This almost defeats the purpose of using Kafka in the first place, and introduces an additional moving part that needs to be maintained. I would suggest that, producer asks Zookeeper for an exclusive partition (or topic+partition) and pass that along with the request in the producer. Handler sees the (topic+)partition to use and sendst he response there. Upon return, release the Zk lock. – Niclas Hedhman Mar 13 '18 at 01:11
1

Easier! You can only write on zookeeper that the UUID X should be answered on partition Y, and make the producer that sent that UUID consume the partition Y... Does that make sense?

SmokeMachine
  • 426
  • 2
  • 5
0

In theory, you could

  1. assign an ID to each request and message that is supposed to get a result message;
  2. create a hash function that would map this ID to an identifier of of a partition,
  3. when sending the result message, use the same hash function to get the identifier of the partition to send it to,
  4. in the producer you could only observe that given partition.

That would reduce the need to crawl many messages in that topic to filter out the result required by the waiting request handler.

Ondra Žižka
  • 43,948
  • 41
  • 217
  • 277
SmokeMachine
  • 426
  • 2
  • 5
0

I think you need a well defined shard key of the service that invokes the request. Your request should contain this shard key and the name of the topic where to post response. Also you should create some sort of state machine and when a message regarding your task comes you transition to some state... this would be for strict async design

Martin Kosicky
  • 471
  • 4
  • 12