52

I got a use case where I need to send key value messages with Kafka Console Producer. So how to achieve this through the Kafka Console Producer command?

Keshav Lodhi
  • 2,641
  • 2
  • 17
  • 23

2 Answers2

100

I found out the solution after some research and the solution is here.

kafka-console-producer command

kafka-console-producer.sh --broker-list localhost:9092 --topic topic-name --property "parse.key=true" --property "key.separator=:"

After running this command you will enter in producer console and from there you can send key, value messages.

For example

key1:value1
key2:value2
key3:value3

For more clarity, I am providing sample key-value message here, emp_info is a key and JSON object is a value.

emp_info: {"emp_id":100,"first_name":"Keshav","last_name":"Lodhi","designation":"DataEngineer"}

Note: Simply sending lines of text will result in messages with null keys. In order to send messages with both keys and values you must set the parse.key and key.separator properties on the command line when running the producer.

Keshav Lodhi
  • 2,641
  • 2
  • 17
  • 23
  • How do I only send 1 message? When I enter the key-value pair, it expects the next line. How do I tell it that I just want to send 1 message? – SAKURA Aug 04 '20 at 20:38
  • 4
    If you want to send only one message, send it from a file which has that one message.. Syntax kafka-console-producer.sh --broker-list localhost:9092 --topic topic-name --property "parse.key=true" --property "key.separator=:" < filePathAndName – Raptor0009 Dec 12 '20 at 14:53
  • In my experience, the `"` characters surrounding the `--property` params are only mandatory if you want to use a special character as the value of the property. In my case I wanted to use `|` as the `key.separator` and I could only make it work like this: `--property "key.separator=|"`. Thanks for this response. – Alex MM Mar 12 '21 at 21:43
  • @SAKURA No, it is not "expecting" the next line. The producer CLI is for sending messages, one per line. When you hit the Enter key, that line will be sent immediately. If you are done, press `Ctrl+C` to terminate the process. You can read this on the official quick start page. – blackr1234 Sep 15 '21 at 18:34
  • 1
    default keySeparator is "\t" (tab). https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L292 – morhook Apr 06 '22 at 18:30
  • @SAKURA just use control+d to finish sending message. – steamfood Apr 20 '22 at 06:53
  • for me didn't work using a ":" separator, I needed to use ";" – adrianosymphony Jul 08 '22 at 01:41
8

By default, the producer doesn’t care about the topic-partition on which the messages are written to and will balance the messages fairly over all the partitions of a topic. Producer picks the partition based on the hash of the record’s key or in round-robin fashion if the record has no key.

  1. Kafka uses the key to specify the target partition. The default
    strategy is to choose a partition based on a hash of the key or use round-robin algorithm if the key is null.

  2. Kafka works with key-value pairs, if key not specified, it will be considered default as null and partition will identified as round-robin fashion.

  3. If we specify key, messages/records with same key goes to same partition

  4. To enable sending full key-value pairs, from the command-line, we need to use two properties as below:

  5. properties

  • parse.key : if it’s true – key is mandatory, by default it’s set as false.

  • key.separator : as below

Examples:

  • key.separator=,
  • key.separator=-
  • key.separator=:

Kafka Console Producer command

kafka-console-producer --broker-list MY-KAFKA:29092 --topic kafka-prod --property parse.key=true --property key.separator=,

Reference: https://shashirl9.medium.com/kafka-producer-internals-d971ac582688

shasr
  • 325
  • 2
  • 5