3

I am trying to connect to a JMX port that I have defined to be 1099 (as is the default) on my Java application (Java 8 and with Spring Boot 1.4 and Tomcat) using a client such as JConsole, Java Mission Control, or Java VisualVM, but I am reaching an error: 

java.rmi.ConnectException: Connection refused to host: 10.xxx.xxx.xx, nested exception is:
     java.net.ConnectException: Connection timed out
     ...

Note that I hid the exact host IP, but it is the pod IP of the particular Kubernetes-managed Docker container that my service is deployed in. I try to connect to the JMX port using the following service URL:

jconsole service:jmx:rmi://<nodeIP>:<nodePort>/jndi/rmi://<nodeIP>:<nodePort>/jmxrmi

I know that JMX opens a random high port, and I have tried to resolve that by including a custom @Configuration class that forces that high port to also serve on port 1099. I have gone into the actual container and pod and ran 

netstat -tulpn

to see the ports that are opened, and I have confirmed that the only ports opened are 8443 (which my application is running on) and 1099 (the JMX port); this indicates that my class works. I also ensured that port 1099 is open on the Kubernetes side, so that is not what is blocking it.

As many answers surrounding a JMX remote connection have suggested, I have tried many variations of the following Java options to no avail:

-Dcom.sun.management.jmxremote  -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=1099 -Dcom.sun.management.jmxremote.jmi.port=1099 -Djava.rmi.server.hostname=0.0.0.0

The answers have suggested forcing the JMX port and the RMI registry port to be the same, but none of these have worked.

I believe the issue may be because the hostname (as I tried to make dynamic with -Djava.rmi.server.hostname=0.0.0.0) can not resolve to the different hostnames (pod IPs) that are created every time my service is deployed.

Thus, it looks like the connection cannot complete because JMX is unable to see what hostname Kubernetes is assigning after my service is deployed.

Is there a way for JMX to recognize the Kubernetes hostname? Or, is there some other way to connect remotely to a JMX port through a Kubernetes-deployed service?

EDIT 1: I have done some additional research, and maybe an optional JMXMP instead of RMI may work? Has anyone gotten this working with Tomcat?

Community
  • 1
  • 1
Tremaine
  • 89
  • 2
  • 9
  • 1
    have you tried using `kubectl port-forward [podname] 1099:1099` to test if it's listening? – 3ocene Dec 09 '16 at 22:25
  • Also, the [10.xxx.xxx.xxx range of IPs are reserved](https://en.wikipedia.org/wiki/Private_network). That isn't a public IP so you likely can't connect to it. You probably need to set up a service and an ingress to talk in to your pod via the node's IP address. – 3ocene Dec 09 '16 at 22:27
  • @3ocene port-forward is not an option in our case. We need to be able to use a node-port for our situation. The 10.xxx.xxx.xxx address is a private IP and the cluster is part of our private network. Note that we did a ssh-tunnel from a node in the cluster to the podIP:1099 and the connection work but it doesn't work with the node:nodePort. – realgenob Dec 09 '16 at 23:15
  • Just to be clear, @realgenob is also working with me and experiencing this same issue. – Tremaine Dec 09 '16 at 23:23
  • Have you tried specifying -Djava.rmi.server.hostname=10.x.x.x? We had a similar setup with Openstack VMs, where OpenStack assigns internal IPs to the VMs, but you use external routed IPs to access the applications. So we specified that parameter with the external IP and it worked (of course then you will have the next problem when you want to make it dynamically). – dunni Dec 09 '16 at 23:47
  • @dunni We have tried that by using a node IP in the cluster in hopes that it would resolve to this particular pod given the node port, but it did not work. – Tremaine Dec 10 '16 at 00:20

1 Answers1

1

The, jmx remote connection is a pain to work with, proxying it is impossible from my view. I had similar problems and in the end I just used jolokia to connect.

Jolokia is a JMX-HTTP bridge giving an alternative to JSR-160 connectors. It is an agent based approach with support for many platforms. In addition to basic JMX operations it enhances JMX remoting with unique features like bulk requests and fine grained security policies. -> http://jolokia.org

Sergiu Vidrascu
  • 246
  • 1
  • 4
  • Thanks for sharing Jolokia. Did you end up installing the agent in the container, then (which it looks like giving an argument to the Java application with the path to the .jar file)? Or, if you had a Spring application context, did you just add a simple Maven dependency? – Tremaine Dec 12 '16 at 18:51
  • In my case I just use the war version. Works very well for my needs. – Sergiu Vidrascu Dec 12 '16 at 19:47
  • Good to hear. I may go down this path if all else fails with the current approach I had; does it involve any continuous monitoring (like through a socket) or is it simply just requests and responses? – Tremaine Dec 12 '16 at 21:17
  • Request and response as far as i used It. – Sergiu Vidrascu Dec 17 '16 at 11:20