kafka-manager not able to read from kafka docker JMX port exposed

Question

I am not able to get my kafka-manager to show messages data/metrics. I am running kafka-manager locally with the below command,

 bin/kafka-manager -Dkafka-manager.zkhosts="localhost:2181"

Also I am checking enable JMX polling on start option.

I am publishing messages to the kafka broker on a topic: test .Kafka-manager view is able to show the topic "test" but does not show the messages count/metrics etc. The kafka-manager application throws an exception which says:

[error] k.m.a.c.OffsetCachePassive - [topic=logstash_topic] An error has occurred while getting topic offsets from broker List((BrokerIdentity(1,kafka-1,9092,9999,false),0))
java.nio.channels.ClosedChannelException: null
at kafka.network.BlockingChannel.send(BlockingChannel.scala:110) ~[org.apache.kafka.kafka_2.11-0.10.0.1.jar:na]
at kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:98) ~[org.apache.kafka.kafka_2.11-0.10.0.1.jar:na]
at kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:83) ~[org.apache.kafka.kafka_2.11-0.10.0.1.jar:na]
at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:149) ~[org.apache.kafka.kafka_2.11-0.10.0.1.jar:na]
at kafka.manager.actor.cluster.OffsetCache$$anonfun$19$$anonfun$21$$anonfun$22.apply(KafkaStateActor.scala:415) ~[kafka-manager.kafka-manager-1.3.3.7-sans-externalized.jar:na]
at kafka.manager.actor.cluster.OffsetCache$$anonfun$19$$anonfun$21$$anonfun$22.apply(KafkaStateActor.scala:412) ~[kafka-manager.kafka-manager-1.3.3.7-sans-externalized.jar:na]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24) ~[org.scala-lang.scala-library-2.11.8.jar:na]
at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24) ~[org.scala-lang.scala-library-2.11.8.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161]
[info] k.m.a.KafkaManagerActor - Updating internal state...
[info] k.m.a.c.BrokerViewCacheActor - Updating broker view...
[info] k.m.a.KafkaManagerActor - Updating internal state...
[error] k.m.j.KafkaJMX$ - Failed to connect to service:jmx:rmi:///jndi/rmi://kafka-1:9999/jmxrmi 
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: kafka-1; nested exception is:
java.net.ConnectException: Operation timed out (Connection timed out)]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) ~[na:1.8.0_161]
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) ~[na:1.8.0_161]
at kafka.manager.jmx.KafkaJMX$.doWithConnection(KafkaJMX.scala:57) ~[kafka-manager.kafka-manager-1.3.3.7-sans-externalized.jar:na]

And my zookeeper and kafka instance are running by docker-compose up -d.

Below is my docker-compose.yml file.

    zookeeper:
    image: confluentinc/cp-zookeeper:latest
    hostname: zookeeper
    environment:
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - "2181:2181"

  kafka-1:
    image: confluentinc/cp-kafka:latest
    hostname: kafka-1
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:9092
      KAFKA_JMX_HOSTNAME: kafka-1
      KAFKA_JMX_PORT: 9999
      KAFKA_JMX_OPTS: "-Djava.rmi.server.hostname=kafka-1 - 
   Dcom.sun.management.jmxremote.local.only=false - 
   Dcom.sun.management.jmxremote.rmi.port=9999 - 
   Dcom.sun.management.jmxremote.port=9999 - 
   Dcom.sun.management.jmxremote.authenticate=false - 
   Dcom.sun.management.jmxremote.ssl=false"
   ports:
      - "9092:9092"
      - "9999:9999"

Really stuck at this one. Appreciate any kind of help. Thanks.

I have set the hostname property in KAFKA_JMX_OPTS for the same. Still no luck. — user7222071, Apr 09 '18 at 06:09
Where are you running the Kafka manager? From another container? Why is it not part of the Compose file? — OneCricketeer, Apr 09 '18 at 12:25
I am running kafka-manager locally from my machine. I have added an entry for kafka-1 in /etc/hosts pointing to container IP of kafka-1. — user7222071, Apr 09 '18 at 13:01
Why do that? It runs perfectly fine in a container. Also, you've exposed 9999 to localhost, so you don't need to touch your hosts file. You used localhost for Zookeeper — OneCricketeer, Apr 09 '18 at 13:04

score 1 · Answer 1 · answered Feb 26 '19 at 06:53

Thank you user7222071, managed to get your code to work after I moved the "-" to the next line for the KAFKA_JMX_OPTS

  kafka-1:
    image: confluentinc/cp-kafka:5.1.2
    volumes:
      - kafka-1:/var/lib/kafka/data
    ports:
      - 19092:19092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper-1:22181,zookeeper-2:32181,zookeeper-3:42181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:19092
      KAFKA_JMX_HOSTNAME: "kafka-1"
      KAFKA_JMX_PORT: 9999
      KAFKA_JMX_OPTS: "-Djava.rmi.server.hostname=kafka-1 
      -Dcom.sun.management.jmxremote.local.only=false 
      -Dcom.sun.management.jmxremote.rmi.port=9999 
      -Dcom.sun.management.jmxremote.port=9999 
      -Dcom.sun.management.jmxremote.authenticate=false 
      -Dcom.sun.management.jmxremote.ssl=false"
    extra_hosts:
      - "moby:127.0.0.1"

  kafka-manager:
    image: kafkamanager/kafka-manager:latest
    ports:
      - 9000:9000
    environment:
      ZK_HOSTS: zookeeper-1:2218

score 0 · Answer 2 · answered Aug 03 '22 at 18:08

FWIW, in my scenario I wanted to monitor Kafka Connect and my connectors specifically -

 connect:
    image: confluentinc/cp-kafka-connect-base:latest
.
.
.
ports:
  - "9999:9999"

.
.
.
environment:
  KAFKA_JMX_HOSTNAME: "127.0.0.1"
  KAFKA_JMX_PORT: 9999
  KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false  -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9999"

Then you can access it using a JMX Client on the Docker Host jmxterm-1.0.2-uber.jar --url 127.0.0.1:9999

kafka-manager not able to read from kafka docker JMX port exposed

2 Answers2