3

I am having a few conceptual mind-blocks. I am looking at using Kafka as below:

---http-requests---> [Web Tier]

[Web Tier] ----composes message and publishes----> [Kafka Topic]

[Kafka Topic] <----consumes message------ [Engine Logic]

[Engine Logic] -----computes, api-calls, publishes result message----> [Kafka Topic]

[Kafka] ---???? somehow get result message to Web Tier---> [Web Tier]

[Web Tier] -----renders HTML and sends http response--->

Using a non-blocking webserver (such as Jetty) the http request will be 'held open' but won't block. Is it possible to use Kafka in a request/response fashion? That is to say, can the Web Tier publish a message to a Topic and then will Kafka know that it must provide a response? Or is it going to be the Web Tier's responsibility to poll a Topic and see if there is a response for it?

I guess what I am asking is what is the best way to use Kafka as an 'interface' upon which the Web Tier and Engine Logic depend so that there is no direct coupling between the Web Tier and the Engine? Thanks.

Zuriar
  • 11,096
  • 20
  • 57
  • 92

1 Answers1

0

I would say that Kafka doesn't fit very naturally into your use case.

It would be the Web Tier's responsibility to poll a topic and see if there is a response for it.

Several problems that I foresee:

  1. Kafka is designed to deliver messages only when requested. In your example, once the data got to the second Kafka Topic. It would sit there until the Web Tier polled for it.
  2. There is no very good way to request a specific message from Kafka, SO Question. In your Web Tier if you polled the Kafka Server, you might get 10 messages, you may get 0. If you get 10 messages, you would have to check and see whether any of those 10 messages are the ones you're looking for. If they're not, you'd have to poll again. Potentially doing that many times depending on how long it took for your particular Engine Logic to complete.

In the SO Question referenced above, in the comments the OP said he went with a Mongo DB so he could query for specific messages.

Community
  • 1
  • 1
Morgan Kenyon
  • 3,072
  • 1
  • 26
  • 38