Stack Overflow
Stack Overflow

Questions tagged [rebalancing]

47 questions
15
votes
1 answer

Kafka Rebalancing and listeners pitfalls

I am reading Kafka: The Definitive Guide and would like to better understand the rebalance listener. The example in the book simple uses a HashMap to maintain the current offsets that have been processed and will commit the current state when a…
Justin Pihony
  • 66,056
  • 18
  • 147
  • 180
11
votes
7 answers

How to check if Kafka Consumer is ready

I have Kafka commit policy set to latest and missing first few messages. If I give a sleep of 20 seconds before starting to send the messages to the input topic, everything is working as desired. I am not sure if the problem is with consumer taking…
Nagireddy Hanisha
  • 1,290
  • 4
  • 17
  • 39
9
votes
2 answers

Rebalancing a table shard, with MySQL/InnoDB

I have a huge InnoDB table (>1TB, >1B rows) that I would like to shard: I would like to make multiple smaller independent tables from that big table. How to do that ? What I've already tried: Moving rows to partitions by SELECTing them from the…
Arnaud Le Blanc
  • 98,321
  • 23
  • 206
  • 194
8
votes
1 answer

Kubernetes - Trigger a rebalancing of pods

I have a kubernetes cluster with a few nodes set up. I want to make sure that pods are distributed efficiently on the nodes. I'll explain: Let's assume that I have two nodes: Node 1 - 2gb ram Node 2 - 2gb ram And I have these pods: Pod 1 - 1gb…
refaelos
  • 7,927
  • 7
  • 36
  • 55
7
votes
2 answers

Kafka connector- cannot stop rebalancing

I am using kafka connector confluent 3.0.1 version.I create a new group named new-group and there are about 20 topics on it.most of these topics is busy.But It is a pity that when I start the connector framework , the system cannot stop rebalanceing…
wuchang
  • 3,003
  • 8
  • 42
  • 66
4
votes
1 answer

Consumer 'group_name' group is rebalancing forever

I am using Kafka: 2.11-1.0.1. The application contains consumers with concurrency=5 for the topic 'X' with partitions=5. When the application is restarted and the message is published on topic 'X' before partition assignment, 5 consumers of topic…
swapnil7
  • 808
  • 1
  • 9
  • 22
4
votes
1 answer

Service Fabric Resource balancer uses stale Reported load

While looking into the resource balancer and dynamic load metrics on Service Fabric, we ran into some questions (Running devbox SDK GA 2.0.135). In the Service Fabric Explorer (the portal and the standalone application) we can see that the balancing…
P. Gramberg
  • 329
  • 1
  • 3
  • 14
3
votes
1 answer

Citus shard rebalancing does not progress

I have a Citus cluster with 3 worker nodes, added a new worker recently then started a rebalance on a table. Everything is OK until here but rebalancing does not end. When I look at get_rebalance_progress() target shard size is always %10 of source…
Kellad
  • 618
  • 1
  • 5
  • 10
2
votes
0 answers

The rebalance function is rebalancing incorrectly so the pop back function cant pop the correct value off

I created a pop_back function and rebalancing for this deque I am making, but when it rebalances, I wanted it to become a b c after the 3rd pop back so that the c can be popped off, therefore, passing the test case. Would this be a rebalance issue…
BadCoder12
  • 21
  • 1
2
votes
1 answer

PostgreSQL Citus Shard Rebalancing after adding new node -> connection to the remote node localhost:5432 failed with the following error

Here I have a problem while applying Citus rebalancing after distributing a specific PostgreSQL table and adding new nodes to scale my database. You can take a look at this useful article if you would like to understand rebalancing in citus before…
Uncle Bent
  • 163
  • 1
  • 1
  • 11
2
votes
0 answers

Is there a way to trigger Kafka Connector rebalancing within a Source Task programatically

Is it possible to trigger Kafka Connector rebalancing from within the Source Task. The task can detect some exceptional conditions which may warrant the connector to rebalance. The SourceTaskContext does not have a method for rebalancing like…
Grizzled DevArch
  • 91
  • 9
2
votes
1 answer

How to add another shard to production for tarantool Database, without downtime?

We use tarantool database (sharded using vshard) in production. We started directly with 4 shards. Now we want to increase it to 6 without downtime. But, after adding two more shards, rebalancer kicks in and it doesn't allow reads/writes to happen.…
jeevan sirela
  • 23
  • 1
  • 7
2
votes
0 answers

Spark-Streaming-Kafka: rebalancing partitions after having used Assign to seek latest offset

I have three Spark Streaming jobs that use ConsumerStrategies.Assign[]() to seek the latest offset that was committed into a database. Each one of these jobs is reading from one of three partitions in a topic (for example: partition 0,1 and 2). If…
Haisam Tarek Elkewidy
  • 97
  • 4
  • 13
2
votes
3 answers

Consumer rebalance while using autocommit

We're using consumer kafka client 0.10.2.0 with the following configuration: props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true"); props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest"); …
Orel
  • 43
  • 7
2
votes
1 answer

Red-Black Tree Insertion Code

I'm attempting to write an implementation of a red-black tree for my own learning and I've been staring at this for a few days now. Could anyone help me figure out how I can get the double rotation cases to work correctly? If you spot anything else…
user3902496
1
2 3 4