4

I have data coming in through RabbitMQ. The data is coming in constantly, multiple messages per second. I need to forward that data to Kafka.

In my RabbitMQ delivery callback where I am getting the data from RabbitMQ I have a Kafka producer that immediately sends the recevied messages to Kafka. My question is very simple. Is it better to create a Kafka producer outside of the callback method and use that one producer for all messages or should I create the producer inside the callback method and close it after the message is sent, which means that I am creating a new producer for each message?

It might be a naive question but I am new to Kafka and so far I did not find a definitive answer on the internet.

EDIT : I am using a Java Kafka client.

mirzaD14
  • 43
  • 4

2 Answers2

4

Creating a Kafka producer is an expensive operation, so using Kafka producer as a singleton will be a good practice considering performance and utilizing resources.

For Java clients, this is from the docs:

The producer is thread safe and should generally be shared among all threads for best performance.

For librdkafka based clients (confluent-dotnet, confluent-python etc.), I can link this related issue with this quote from the issue:

Yes, creating a singleton service like that is a good pattern. you definitely should not create a producer each time you want to produce a message - it is approximately 500,000 times less efficient.

ndogac
  • 1,185
  • 6
  • 15
  • A question. If I am creating a singleton instance of Kafka producer (say in my test framework), when and how do I close it? To give some context, I am using Junit5 and there could be say 10 test classes each having multiple tests that use the same producer. If I use the @AfterAll method, it will close the producer after first test class finishes. So that would not work. Or would it automatically close at the end of test run? – Pramod Yadav Jun 20 '23 at 15:09
  • @PramodYadav I think this should be a question instead of a comment. One suggestion you can consider in this scenario is to use a counter in your test class to keep track of the producer usage. When the counter gets to `0` in your `AfterAll` method, you can close the producer. Remember to cover error handling (what if `createProducer` fails?) and handle concurrency (`usageCount` variable needs to be safely incremented and decremented across threads). – ndogac Jun 20 '23 at 18:58
  • Thanks for your reply @ndogac. I found an answer for this question here: https://stackoverflow.com/questions/43282798/in-junit-5-how-to-run-code-before-all-tests/65897949?noredirect=1#comment135156634_65897949 – Pramod Yadav Jul 11 '23 at 10:50
0

Kafka producer is stateful. It contains meta info(periodical synced from brokers), send message buffer etc. So create producer for each message is impracticable.

louxiu
  • 2,830
  • 3
  • 28
  • 42