I already have a kafka consumer group running in multiple instances. I have a requirement to pause the consumption of a particular topic whenever I reach a threshold. When I try to pause the kafka consumer group for a topic, the entire group gets paused. As per documentation, we can pause topics on a consumer level. However, when I create a consumer, it is not consuming any message. The options I used for the consumer are:
const kafkaClient = new kafka.KafkaClient({
kafkaHost : this.brokerUrl
})
{
kafkaHost: this.brokerUrl,
groupId: "registeredGroupId"
}
const consumer = new kafka.Consumer(
kafkaClient,
topics,
consumerOptions
);
Do I also need to initialize a consumer group with the group id? Will the consumer then be activated and pause topic based consumption? I tried running consumer and consumer group simultaneously but only the consumer group receives the message. If I don't create a consumer group, then the group shows inactive in kafka consumers list.
Is there a way to have the same consumer group run on 3 instances as it is running currently and add consumers in each of the instance which would pause the message consumption topic wise?
