1

I'm trying to get JMX to work with spotify/kafka Docker image.

I have a docker-compose.yml with:

version: '2'
services:
  my-kafka:
    image: spotify/kafka # Using this instead of wurstmeister's because it didn't work with /var/run/docker.sock on Windows while this one has integrated ZK and works
    hostname: my-kafka
    restart: unless-stopped
    ports:
      - "9092:9092" # Kafka
      - "2181:2181" # Zookeeper
      - "7203:7203" # JMX
    environment:
      KAFKA_ADVERTISED_HOST_NAME: my-kafka
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_OPTS: "-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.rmi.port=7203 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=my-kafka"

I then try to test it using jconsole my-kafka:7203 and getting a 'Connection failed: error during JRMP connection establishment; nested exception is: java.io.EOFException.' immediately. By contrast, using a non-existent host leads to "Unknown host" and a different port to "Connection refused".

docker ps shows:

692eb6659aba        spotify/kafka                 "supervisord -n"         10 minutes ago      Up 10 minutes               0.0.0.0:2181->2181/tcp, 0.0.0.0:7203->7203/tcp, 0.0.0.0:9092->9092/tcp                       docker_my-kafka_1

ps aux within the container shows:

root        11  1.8  3.3 5718844 274608 ?      Sl   06:16   0:12 /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true -Xloggc:/opt/kafka_2.11-0.10.1.0/bin/../logs/kafkaServer-gc.log -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false ...offtopic skipped... -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.rmi.port=7203 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=my-kafka kafka.Kafka /opt/kafka_2.11-0.10.1.0/config/server.properties

The container log is boring without JMX being even mentioned, ends with "kafka entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)".

There is a related question kafka monitoring using JMX when running kafka in docker in boot2docker but I'd prefer to stick to spotify/kafka so that ZK & Kafka live in the same container.

ashleedawg
  • 20,365
  • 9
  • 72
  • 105
John M
  • 1,469
  • 17
  • 41

4 Answers4

0

Can you try specifying the host address(where you are running the docker run) in the property : 

-Djava.rmi.server.hostname=xx.xx.xx.xx
Rambler
  • 4,994
  • 2
  • 20
  • 27
  • The result is exactly the same as before: Connection failed: error during JRMP connection establishment; nested exception is: java.io.EOFException. If it would worked, getting this into the docker-compose.yml file would also be a hassle. – John M Jan 15 '17 at 19:40
0

jmx options are already present in the starting script /opt/kafka_2.11-0.10.1.0/bin/kafka-run-class.sh of the spotify/kafka docker image. You can grep them from inside the container

grep jmx /opt/kafka_2.11-0.10.1.0/bin/kafka-run-class.sh

and you see them at the beginning of java options of the jvm process inside the container

ps -ef java

So, it's enough you set JMX_PORT environment variable in docker-compose.yml and you remove additional (repeated) KAFKA_OPTS.

version: '2'
services:
  my-kafka:
    image: spotify/kafka # Using this instead of wurstmeister's because it didn't work with /var/run/docker.sock on Windows while this one has integrated ZK and works
    hostname: my-kafka
    restart: unless-stopped
    ports:
      - "9092:9092" # Kafka
      - "2181:2181" # Zookeeper
      - "7203:7203" # JMX
    environment:
      KAFKA_ADVERTISED_HOST_NAME: my-kafka
      KAFKA_ADVERTISED_PORT: 9092
      JMX_PORT: 7203

Then after starting a new container and having resolved my-kafka in your host, jconsole my-kafka:7203 will connect to the container's jmx port.

gile
  • 5,580
  • 1
  • 25
  • 31
  • You are very right about the duplicated options, but for some reason 'jconsole' still doesn't connect - now it just hangs upon connection :/. – John M Jan 15 '17 at 21:47
  • I tried on my pc (Ubuntu Linux + Oracle JDK v. 1.8) and it works. Check your jconsole / jdk version. The container runs 1.8.x. – gile Jan 15 '17 at 22:02
  • `1.8.0_112-b16` Oracle JDK on MacOS X where I run `jconsole`, OpenJDK `1.8.0_111-8u111-b14-2~bpo8+1-b14` in the container. How do you run `jconsole` and what do you fill out? I don't need a username/password, do I? – John M Jan 16 '17 at 05:29
  • My Cassandra container runs exact same `1.8.0_111-8u111-b14-2~bpo8+1-b14` and JMX works on it fine (admittedly, I created a `jmxremote.password` file and am using `com.sun.management.jmxremote.authenticate=true` in it so it's different). Just that I don't think it's the JDK incompatibility as I'd hit it then as well. – John M Jan 16 '17 at 05:32
  • I run "jconsole 172.20.0.2:7203" where 172.20.0.2 is the container IP get from "docker inspect kafka_my-kafka_1". But docker networking differs between linux and macos. Anyway I get a message from jconsole"Secure connection failed. Retry insecurely?" and after selkecting "Insecure connection" it works. Why you don't tru jvisualvm instead of jconsole? If you run jvisualvm the add remote host and then add JMX remote connection, you can select insecure connection. May be it's better in your environment. – gile Jan 16 '17 at 08:10
0

Docker containers run on an IP set within your machine. To see these, run:

docker inspect docker_my-kafka_1

This will output the IP that the machine is working on, something like:

"IPAddress": "172.17.0.2",

You will need to add the Ip found above to:

-Djava.rmi.server.hostname=$INSERT_IP_HERE

Marc Murray
  • 76
  • 2
  • That means regenerating the docker-compose.yml file every time I switch development environments. That doesn't sound a good approach. – John M Jan 23 '17 at 16:48
  • Unfortunately, this is the way you have to do it if you have no configuration management in place. – Marc Murray Jan 24 '17 at 17:14
0

As you prefer to use a solution where both ZK & Kafka live in the same container, have a look at https://github.com/Landoop/fast-data-dev

This docker is focusing on providing an excellent Kafka Development Environment, so among others it exposes all JMXs

Antonios Chalkiopoulos
  • 860
  • 1
  • 9
  • 10