119

Is there any way I can view the message content sent to kafka for a given topic? Say some thing like view last 5 messages for this topic, if that it possible.

funnydman
  • 9,083
  • 4
  • 40
  • 55
Sandy
  • 2,253
  • 6
  • 24
  • 33
  • 1
    [KaDeck](https://www.kadeck.com) is a free "Kafka Topic/Message Browser" which runs on Mac, Linux and Windows as Desktop Application. It also supports Avro (Schema Reg + Embedded), Json, String, etc... If you need a webservice you need to go for the enterprise edition though. – Ben May 29 '19 at 12:47
  • 2
    Upvote the `kcat` answer for simplicity – Clintm Jan 13 '23 at 16:01

8 Answers8

194

You can use console consumer to view messages produced on some topic:

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
Andrew Theken
  • 3,392
  • 1
  • 31
  • 54
vatsal mevada
  • 5,148
  • 7
  • 39
  • 68
  • This command also required --zookeeper details , I added those like - ./kafka-console-consumer.sh --zookeeper localhost:2181 --bootstrap-server localhost:9092 --topic test. But now I get - kafka.common.ConsumerRebalanceFailedException: console-consumer-18088_sp-1496052524018-50bb9338 can't rebalance after 4 retries at kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:660) at – Sandy May 29 '17 at 10:11
  • 1
    Which is a separate issue. Take a look at this: https://stackoverflow.com/questions/22563401/rebalancing-issue-while-reading-messages-in-kafka – vatsal mevada May 29 '17 at 10:15
  • 2
    Use bin/kafka-console-consumer.sh --new-consumer --bootstrap-server localhost:9092 --topic test --from-beginning – Luciano Afranllie May 29 '17 at 12:27
  • 3
    `bootstrap-server` vs. `zookeeper` is a matter of kafka version. – Joe Atzberger Sep 14 '17 at 23:07
  • WARN [Consumer clientId=consumer-1, groupId=console-consumer-22072] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) – Alexey Sh. Jul 02 '20 at 07:23
  • @vatsalmevada In my case I see the warnings: WARN [Consumer clientId=consumer-console-consumer-85244-1, groupId=console-consumer-85244] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient) ^CProcessed a total of 0 messages But we have SASL PLAIN authentication enabled (without SSL), can it be the reason? and how connection has to be established in that case? – Uladzimir Sharyi Aug 16 '21 at 09:54
  • For windows .bin\windows\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic topic-name --from-beginning – Prashanth Debbadwar Mar 10 '22 at 11:15
  • 1
    Checkout `kcat` – Clintm Jan 13 '23 at 16:01
24

On server where your admin run kafka find kafka-console-consumer.sh by command find . -name kafka-console-consumer.sh then go to that directory and run for read message from your topic

./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning --max-messages 10

note that in topic may be many messages in that case I use --max-messages key

Mikhail
  • 729
  • 1
  • 8
  • 16
4

Use the Kafka consumer provided by Kafka :

bin/kafka-console-consumer.sh --bootstrap-server BROKERS --topic TOPIC_NAME

It will display the messages as it will receive it. Add --from-beginning if you want to start from the beginning.

ImbaBalboa
  • 851
  • 9
  • 23
2

If you're wondering why the original answer is not working. Well it might be that you're not in the home directory. Try this:

$KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
Haim
  • 61
  • 1
  • 7
2

A simple terminal-based tool you can use is kcat.

# Display values
kcat -C -b <bootstrap_server_host:port> -t <topic>

# Display key and values separated by ':'
kcat -C -K : -b <bootstrap_server_host:port> -t <topic>

Note however that deserialization options are a bit limited since it doesn't support protobuf, but it does support Avro.

frblazquez
  • 115
  • 1
  • 10
1

If you doing from windows folder, I mean if you are using the kafka from windows machine

kafka-console-consumer.bat --bootstrap-server localhost:9092 --<topic-name> test --from-beginning
Kuldeep Kumar
  • 754
  • 1
  • 7
  • 12
1

For me the easiest way is, if you are using a JetBrains IDE (I don't know if the others has that plugin), download Big Data Tools plugin.

It will be added to right toolbar. You can connect to your Kafka Server. If it is on localhost the URL of the server is set to default to 127.0.0.1:9092.

After connected successfully, the Kafka will be added your bottom toolbar. Click it and click the Add Consumer button. Select your topic to consume, and start consuming.

Umut Emre Önder
  • 98
  • 11
0

Old version includes kafka-simple-consumer-shell.sh (https://kafka.apache.org/downloads#1.1.1) which is convenient since we do not need cltr+c to exit.

For example

kafka-simple-consumer-shell.sh --broker-list $BROKERIP:$BROKERPORT --topic $TOPIC1 --property print.key=true --property key.separator=":"  --no-wait-at-logend
Gautam Savaliya
  • 1,403
  • 2
  • 20
  • 31
bhargav joshi
  • 329
  • 3
  • 6