I'm trying to connect the ZABBIX Java Gateway to my Kafka server through JMX. I receive a connection time out:
zabbix-java-gateway_1 | 2020-12-21 21:02:20.181 [pool-1-thread-4] WARN com.zabbix.gateway.SocketProcessor - error processing request, item "jmx.discovery[beans,"kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topic=*"]" failed: Connection timed out: service:jmx:rmi:///jndi/rmi://192.168.99.100:12345/jmxrmi
Error in the GUI:
Kafka configuration
For Kafka I’m using the Wurstmeister Docker image. The kafka service in docker-compose.yml looks as follows:
kafka:
build:
context: .
dockerfile: _docker/kafka/Dockerfile
container_name: kafka
env_file:
- kafka/kafka.env
environment:
JMX_PORT: 12345
KAFKA_JMX_OPTS: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=12345
ports:
- "9092:9092"
- "12345:12345"
depends_on:
- zookeeper
volumes:
- ./kafka/volumes/data:/kafka
networks:
- streaming
This my my .env file for Kafka:
KAFKA_BROKER_ID=1
KAFKA_LISTENERS="INTERNAL://0.0.0.0:19092,EXTERNAL://0.0.0.0:9092"
KAFKA_ADVERTISED_LISTENERS="INTERNAL://kafka:19092,EXTERNAL://localhost:9092"
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP="INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT"
KAFKA_INTER_BROKER_LISTENER_NAME="INTERNAL"
KAFKA_ZOOKEEPER_CONNECT="zookeeper:2181"
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR="1"
# topic-name:partition:replication_factor:cleanup.policy:retention.ms
KAFKA_CREATE_TOPICS="iot-parser-errors:1:1:delete:7889400000,iot-raw-data:1:1:delete:604800017,iot-parsed-data:1:1:delete:2629800000,soc-data:1:1:delete:1209600033,battery-data:1:1:delete:31557600000"
KAFKA_LOG4J_LOGGERS="kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
JMX_PORT=12345
KAFKA_JMX_OPTS="-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=127.0.0.1 -Dcom.sun.management.jmxremote.rmi.port=12345"
ZABBIX configuration
I have set up zabbix using the the official zabbix docker image for Ubuntu with MySQL and Nginx. I copied this docker-compose file.
I have uncommented the following environment variables in .env_srv:
ZBX_JAVAGATEWAY_ENABLE=true
ZBX_JAVAGATEWAY=zabbix-java-gateway
ZBX_JAVAGATEWAYPORT=10052
ZBX_STARTJAVAPOLLERS=5
This is what my .env_java file looks like (I have increased the time out to 30):
ZBX_START_POLLERS=5
ZBX_TIMEOUT=30
# Possible values: trace, debug, info, want, error, all, off
ZBX_DEBUGLEVEL=debug
This is how I have set up the JMX interface in the zabbix GUI. I’m using the Apache Kafka by JMX template.
ZABBIX log
zabbix-java-gateway_1 | 2020-12-21 20:46:05.055 [pool-1-thread-4] DEBUG com.zabbix.gateway.SocketProcessor - starting to process incoming connection
zabbix-java-gateway_1 | 2020-12-21 20:46:05.056 [pool-1-thread-4] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading Zabbix protocol header
zabbix-java-gateway_1 | 2020-12-21 20:46:05.056 [pool-1-thread-4] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading 8 bytes of data length
zabbix-java-gateway_1 | 2020-12-21 20:46:05.056 [pool-1-thread-4] DEBUG c.z.gateway.BinaryProtocolSpeaker - reading 247 bytes of request data
zabbix-java-gateway_1 | 2020-12-21 20:46:05.056 [pool-1-thread-4] DEBUG c.z.gateway.BinaryProtocolSpeaker - received the following data in request: {"request":"java gateway jmx","username":"zabbix","password":"zabbix","jmx_endpoint":"service:jmx:rmi:///jndi/rmi://192.168.99.100:12345/jmxrmi","keys":["jmx[\"kafka.network:type=RequestMetrics,name=TemporaryMemoryBytes,request=Fetch\",\"Max\"]"]}
zabbix-java-gateway_1 | 2020-12-21 20:46:05.056 [pool-1-thread-4] DEBUG com.zabbix.gateway.SocketProcessor - dispatched request to class com.zabbix.gateway.JMXItemChecker
zabbix-java-gateway_1 | 2020-12-21 20:46:05.056 [pool-1-thread-4] DEBUG c.z.g.ZabbixJMXConnectorFactory - connecting to JMX agent at 'service:jmx:rmi:///jndi/rmi://192.168.99.100:12345/jmxrmi'
zabbix-web-nginx-mysql_1 | 172.31.0.1 - - [21/Dec/2020:21:46:10 +0100] "POST /zabbix.php?action=notifications.get&sid=fa4abac3a2f8629e&output=ajax HTTP/1.1" 200 436 "http://localhost:8081/hosts.php" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36" "-"
zabbix-web-nginx-mysql_1 | 172.31.0.1 - - [21/Dec/2020:21:46:11 +0100] "POST /jsrpc.php?output=json-rpc HTTP/1.1" 200 74 "http://localhost:8081/hosts.php" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36" "-"
zabbix-web-nginx-mysql_1 | 127.0.0.1 - - [21/Dec/2020:21:46:12 +0100] "GET / HTTP/1.1" 200 3624 "-" "curl/7.68.0" "-"
zabbix-web-nginx-mysql_1 | 127.0.0.1 - - [21/Dec/2020:21:46:22 +0100] "GET / HTTP/1.1" 200 3624 "-" "curl/7.68.0" "-"
zabbix-web-nginx-mysql_1 | 172.31.0.1 - - [21/Dec/2020:21:46:22 +0100] "POST /jsrpc.php?output=json-rpc HTTP/1.1" 200 74 "http://localhost:8081/hosts.php" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36" "-"
zabbix-web-nginx-mysql_1 | 127.0.0.1 - - [21/Dec/2020:21:46:32 +0100] "GET / HTTP/1.1" 200 3624 "-" "curl/7.68.0" "-"
zabbix-web-nginx-mysql_1 | 172.31.0.1 - - [21/Dec/2020:21:46:33 +0100] "POST /jsrpc.php?output=json-rpc HTTP/1.1" 200 74 "http://localhost:8081/hosts.php" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36" "-"
zabbix-java-gateway_1 | 2020-12-21 20:46:35.056 [pool-1-thread-4] WARN com.zabbix.gateway.SocketProcessor - error processing request, item "jmx["kafka.network:type=RequestMetrics,name=TemporaryMemoryBytes,request=Fetch","Max"]" failed: Connection timed out: service:jmx:rmi:///jndi/rmi://192.168.99.100:12345/jmxrmi
zabbix-java-gateway_1 | 2020-12-21 20:46:35.057 [pool-1-thread-4] DEBUG com.zabbix.gateway.SocketProcessor - error caused by
zabbix-java-gateway_1 | com.zabbix.gateway.ZabbixException: Connection timed out: service:jmx:rmi:///jndi/rmi://192.168.99.100:12345/jmxrmi
zabbix-java-gateway_1 | at com.zabbix.gateway.JMXItemChecker.getValues(JMXItemChecker.java:132) ~[zabbix-java-gateway-5.2.2.jar:na]
zabbix-java-gateway_1 | at com.zabbix.gateway.SocketProcessor.run(SocketProcessor.java:79) ~[zabbix-java-gateway-5.2.2.jar:na]
zabbix-java-gateway_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_275]
zabbix-java-gateway_1 | at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_275]
zabbix-java-gateway_1 | at java.lang.Thread.run(Thread.java:748) [na:1.8.0_275]
zabbix-java-gateway_1 | 2020-12-21 20:46:35.057 [pool-1-thread-4] DEBUG c.z.gateway.BinaryProtocolSpeaker - sending the following data in response: {"response":"failed","error":"Connection timed out: service:jmx:rmi:\/\/\/jndi\/rmi:\/\/192.168.99.100:12345\/jmxrmi"}
zabbix-java-gateway_1 | 2020-12-21 20:46:35.057 [pool-1-thread-4] DEBUG com.zabbix.gateway.SocketProcessor - finished processing incoming connection
Kafka log
And here are the logs of the kafka container. I don’t understand why the port 12345 seems to be in use.
zabbix-java-gateway_1 | at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_275]
kafka | Error: JMX connector server communication error: service:jmx:rmi://952116a91006:12345
kafka | sun.management.AgentConfigurationError: java.rmi.server.ExportException: Port already in use: 12345; nested exception is:
kafka | java.net.BindException: Address in use (Bind failed)
kafka | at sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:800)
kafka | at sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:468)
kafka | at sun.management.Agent.startAgent(Agent.java:262)
kafka | at sun.management.Agent.startAgent(Agent.java:452)
kafka | Caused by: java.rmi.server.ExportException: Port already in use: 12345; nested exception is:
kafka | java.net.BindException: Address in use (Bind failed)
kafka | at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:346)
kafka | at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:254)
kafka | at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411)
kafka | at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
kafka | at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:237)
kafka | at sun.management.jmxremote.ConnectorBootstrap$PermanentExporter.exportObject(ConnectorBootstrap.java:199)
kafka | at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:146)
kafka | at javax.management.remote.rmi.RMIJRMPServerImpl.export(RMIJRMPServerImpl.java:122)
kafka | at javax.management.remote.rmi.RMIConnectorServer.start(RMIConnectorServer.java:404)
kafka | at sun.management.jmxremote.ConnectorBootstrap.exportMBeanServer(ConnectorBootstrap.java:796)
kafka | ... 3 more
kafka | Caused by: java.net.BindException: Address in use (Bind failed)
kafka | at java.net.PlainSocketImpl.socketBind(Native Method)
kafka | at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:387)
kafka | at java.net.ServerSocket.bind(ServerSocket.java:375)
kafka | at java.net.ServerSocket.<init>(ServerSocket.java:237)
kafka | at java.net.ServerSocket.<init>(ServerSocket.java:128)
kafka | at sun.rmi.transport.proxy.RMIDirectSocketFactory.createServerSocket(RMIDirectSocketFactory.java:45)
kafka | at sun.rmi.transport.proxy.RMIMasterSocketFactory.createServerSocket(RMIMasterSocketFactory.java:345)
kafka | at sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666)
kafka | at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:335)
kafka | ... 12 more
JMX seems to be working fine
On the Kafka side, JMX seems to be working correctly because I can establish a connection via jconsole 127.0.0.1:12345. The image below shows the Kafka MBeans.
What could be causing this time out?