How does RD_KAFKA_PARTITION_UA work in librdkafka?

Question

I have a producer in php which publishes to queue. I am using php-rdkafka library for the same.

Following is the code:

        $conf->set('log_level', LOG_DEBUG);
        $conf->set('debug', 'all');
        $rk = new RdKafka\Producer($conf);
        $rk->addBrokers("34.93.118.165:9092");
        $topic = $rk->newTopic("download_first");
        $topic->produce(RD_KAFKA_PARTITION_UA, 0, utf8_encode($message));
        $rk->flush(100);

and I have 2 python consumers consuming from the queue.

The topic has 10 partitions. Most of the times when a new message is produced it goes to the same partition as before which is being consumed by a single consumer and as a result other consumer sits idle.

Is there any other partitioner assigning thing I can use while producing messages?

Have you tried reading the librdkafka documentation on config variables? — OneCricketeer, Jun 10 '20 at 14:49
@OneCricketeer Yes, all it says is that using the above argument (RD_KAFKA_PARTITION_UA) lets librdkafka decide the partition but doesn't say anything about how — Rohit Aggarwal, Jun 11 '20 at 05:14

Rajat Sharma · Accepted Answer · 2020-06-12T11:11:06.720

4

https://github.com/arnaud-lb/php-rdkafka uses librdkafka underneath which is a C based Kafka library.

As per the docs RD_KAFKA_PARTITION_UA is for using automatic partitioning using the topic's partitioner function or you can pass the exact partition number instead of RD_KAFKA_PARTITION_UA to force a partition.

Or you call $conf->setPartitioner(RD_KAFKA_MSG_PARTITIONER_RANDOM); for enabling random partitioner function. Here's the list of possible constants for setPartitioner

edited Jun 12 '20 at 11:11

answered Jun 12 '20 at 10:59

Rajat Sharma

86
5

Thanks for the answer. It worked fine after adding additional configuration ($conf->set('partitioner','random');) – Rohit Aggarwal Jun 12 '20 at 13:09

How does RD_KAFKA_PARTITION_UA work in librdkafka?

1 Answers1