Spark-Streaming-Kafka: rebalancing partitions after having used Assign to seek latest offset

Asked Mar 15 '18 at 00:54

Active Nov 15 '18 at 13:34

Viewed 658 times

I have three Spark Streaming jobs that use ConsumerStrategies.Assign[]() to seek the latest offset that was committed into a database.

Each one of these jobs is reading from one of three partitions in a topic (for example: partition 0,1 and 2). If one of the Spark Streaming jobs fails, is it possible to rebalance that partition to one of the other two jobs that are currently running.

I know you can do that in normal Kafka using ConsumerRebalanceListener, onPartitionsProvoked() and onPartitionsAssigned(). But how would you do that in Spark-Streaming-Kafka?

edited Mar 15 '18 at 12:53

pheeleeppoo

1,491
6
25
29

asked Mar 15 '18 at 00:54

Haisam Tarek Elkewidy

Spark-Streaming-Kafka: rebalancing partitions after having used Assign to seek latest offset

0 Answers0