2

When configuring a consumer, you can specify:

group.instance.id - "A unique identifier of the consumer instance provided by the end user. Only non-empty strings are permitted. If set, the consumer is treated as a static member, which means that only one instance with this ID is allowed in the consumer group at any time. This can be used in combination with a larger session timeout to avoid group rebalances caused by transient unavailability (e.g. process restarts). If not set, the consumer will join the group as a dynamic member, which is the traditional behavior."

or client.id - "An id string to pass to the server when making requests. The purpose of this is to be able to track the source of requests beyond just ip/port by allowing a logical application name to be included in server-side request logging."

for my use case, i needed the exact behavior in the group.instance.id description, but I am wondering why there are two different configuration properties, and it took some time to figure out that client.id is insufficient.

Why are there two?

aran
  • 10,978
  • 5
  • 39
  • 69
Dave Ankin
  • 1,060
  • 2
  • 9
  • 20

1 Answers1

4

They are totally different parameters, I've also had to re-read the docs in order to get this clear.

  • client.id is an unique identifier of the consumer, which is mainly used for tracking the source of the requests and log them.
  • group.instance.id is an unique identifier of the consumer used to make the consumer static, and is not related to client.id.

That means you could have these two values in your consumer A's configuration, for example:

//Consumer A
client.id = 123
group.instance.id = 1

The key here is within the static membership paradigm introduced in KIP-345.

The idea is to avoid the partition changes between consumer threads after a rebalance. For example, if you have three different consumers from the same group.id, and set them unique group.instance.ids, each of those consumers will be assigned the same partitions, always.

For example, consider three consumers inside the same group:

Consumer A 
- group.instance.id = A
Consumer B 
- group.instance.ud = B
Consumer C
- group.instance.ud = C

When reading from a Kafka topic with 9 partitions, they will ALWAYS hold the same partitions assigned, for example:

A (0,1,2) - B (3,4,5) - C (6,7,8)

enter image description here

WHat static membership avoids is the partition change caused by the default range assignor, which works by sorting the members in the group and then assigning partition ranges to achieve balance.

This is well explained in this doc (do not confuse member.id with client.id) enter image description here

  1. In red, Consumers without group.instance.id have to change partitions with each rebalance due to the assignor.
  2. In green, consumers that did set an instanceId(group.instance.id), becoming static consumers. static membership makes each consumer "master" of it initial set of partitions. Even if a rebalance occurs, each consumer always holds the same partitions, avoiding partition changes between consumer threads.

TLDR;

In summary, those parameters are not related and the idea of setting a group.instance.id is to make the consumer static:

The idea is summarized as static membership, which in contrary to dynamic membership (the one our system currently uses), is prioritizing "state persistence" over "liveness". The idea of this KIP is to reduce number of rebalances by introducing a new concept called static membership

aran
  • 10,978
  • 5
  • 39
  • 69
  • thanks, this is very helpful, member.id is next on my list to understand. the use case is ephemeral consumer - needs to connect, catch up, and exit - like kcat – Dave Ankin Jul 31 '22 at 03:48
  • 1
    do not care about `member.id`. As per the docs: 'In the current architecture, during each rebalance consumer groups on broker side will assign new member a randomly generated id called `member.id`.' So is not relevant or can be changed. Is just a random number given to new members of the group – aran Jul 31 '22 at 03:59