2

Kafka uses math.abs(key.hashCode) % numPartitions to calculate the partition to send.

What if hashCode is Integer.MIN_VALUE?

As math.abs(Integer.MIN_VALUE) is a negative number, kafka would send to a negative partition. How is that handled, should I care about this?

Frederic A.
  • 3,504
  • 10
  • 17
ntysdd
  • 1,206
  • 2
  • 9
  • 19
  • As you said, it use the math.abs() first, so why you worry about this case? – GuangshengZuo Aug 25 '17 at 03:09
  • It's not a bad question: try than in the repl: `math.abs(Integer.MIN_VALUE)` prints: `res0: Int = -2147483648` – Frederic A. Aug 25 '17 at 03:11
  • related: https://stackoverflow.com/questions/5444611/math-abs-returns-wrong-value-for-integer-min-value#5444634 – Frederic A. Aug 25 '17 at 03:17
  • @FredericA. oh, thanks for your share! – GuangshengZuo Aug 25 '17 at 03:18
  • Too bad scala's doc says abs results >= 0 – ntysdd Aug 25 '17 at 03:20
  • 1
    What is the question about?? it should go to `math.abs(Int.MaxValue) % numPartitions` partition = `2147483647 % numberOfPartitions` but where did you find that code btw I see following code for partitioning `Utils.toPositive(nextValue) % availablePartitions.size()`. where toPositive is `number & 0x7fffffff` – prayagupa Aug 25 '17 at 03:24
  • @prayagupd see the edit – ntysdd Aug 25 '17 at 03:27
  • Sorry it has to be `math.abs(Int.MinValue) % numPartitions = 0th partition` above. And `Int.MinValue & 0x7fffffff` gives you 0, `-1 & 0x7fffffff = 2147483647` – prayagupa Aug 25 '17 at 03:37

1 Answers1

1

Actually, when I search this in kafka code, It does not use the math.abs() to convert the negative to positive.

It use that:

public static int toPositive(int number) {
    return number & 0x7fffffff;
}

So It can solve the problem you worried even if the number is 2147483648, it will be converted to 0

A cheap way to deterministically convert a number to a positive value. When the input is positive, the original value is returned. When the input number is negative, the returned positive value is the original value bit AND against 0x7fffffff which is not its absolutely value.

GuangshengZuo
  • 4,447
  • 21
  • 27
  • really? Which version? – ntysdd Aug 25 '17 at 03:41
  • this is the java kafka client which is also in kafka codebase. You can find in https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/utils/Utils.java. this is not new feature.. – GuangshengZuo Aug 25 '17 at 03:45
  • And the defaultPartitioner is in https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java – GuangshengZuo Aug 25 '17 at 03:47
  • I was looking into an old version of scala DefaultPartitioner – ntysdd Aug 25 '17 at 04:02