5

I want to give some time to my consumer to restart so that unnecessary rebalance doesnt happen. How can I do that? In case of shutdown, I want replication to come in picture and after some time if consumer is not back up, rebalance should occur else not.

R.B.
  • 479
  • 5
  • 19
  • I have 3 instances of consumer groups. 1 goes down, data starts copying on other 2 machines immediately without wait. In some acceptable time, original instance comes back up. This movement of data is unnecessary for me. If the time elapsed is greater than configurable acceptable time, data movement should start. – R.B. Jun 12 '19 at 11:41
  • I think that "static group membership" is what you are looking for. It's part of upcoming Kafka 2.3 release. For details, see the corresponding KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances – Matthias J. Sax Jun 12 '19 at 17:38
  • Is the 2.3 release going as per plan or is there any change in schedule? – R.B. Jun 13 '19 at 09:11
  • The vote is already in progress. If no blocker is found, 2.3 should be released soon. – Matthias J. Sax Jun 13 '19 at 16:26

1 Answers1

4

There's broker level config called group.initial.rebalance.delay.ms you can tweak.

The amount of time the group coordinator will wait for more consumers to join a new group before performing the first rebalance. A longer delay means potentially fewer rebalances, but increases the time until processing begins.

https://kafka.apache.org/documentation/

Levani Kokhreidze
  • 125
  • 7
  • 4
    This config only works if you create a new consumer group, and the delay is only considered when the first consumer joins an empty group. – Matthias J. Sax Jun 12 '19 at 17:33
  • @MatthiasJ.Sax per the Wiki, https://cwiki.apache.org/confluence/display/KAFKA/KIP-134%3A+Delay+initial+consumer+group+rebalance, this setting applies to "the initial rebalance of a new, or empty, consumer group". I realize it does not help the OP here, since it does not apply to groups with consumers, but it implies an existing group with no members would also benefit from the delay. Is that your understanding? Has that changed since this comment in 06/2019? – ammills01 Nov 02 '20 at 22:08
  • Nothing about it changed so far. -- Not that the config name contains `initial` and the description of the parameter also says: to join a _new_ group. -- I am not totally sure atm, if it applied to an existing but empty group. – Matthias J. Sax Nov 03 '20 at 01:55