Multiple kafka producer threads getting spawn

Question

I am new to kafka. I have two kafka brokers and I am trying to push data through these two brokers. One is primary and other one is backup.

I am doing a small analysis in which I am pushing data to kafka queue through a thread pool executor. While doing so, I kept max pool size as 1 and size of array blocking queue as 2. I triggered 10 requests through Jmeter and 7 of these requests went to rejection handler (as expected) and 3 went to be processed for kafka queue.

Took a thread dump to analyse state of kafka threads, 4 producer threads were spun up for kafka producer.

I could not understand this as I am using two brokers and here 3 messages are getting processed through kafka (2 in blocking queue, 1 in thread), then how 4 producer threads got spun up?

PS : I cannot share code piece here due to security concerns.

Answer 1

I could not understand this as I am using two brokers and here 3 messages are getting processed through kafka (2 in blocking queue, 1 in thread), then how 4 producer threads got spun up?

My general feeling is that you shouldn't have to worry about how many threads the Kafka API uses. Threads are relatively cheap and the Kafka code is just trying to be efficient with its communications. If you have a single Kafka Producer object then I'm surprised there are 4 threads but it may be making a connection to each broker and may have a read and write thread for each. Looking at the stack trace for each thread might cue you into what they are doing. You can also look at the Kafka code might also help.

I triggered 10 requests through Jmeter and 7 of these requests went to rejection handler (as expected) and 3 went to be processed for kafka queue.

Typically I add a rejection handler which blocks. See my answer here. This will block your request handler depending on how many threads are in your pool.