multiple app nodes how to expose jmx in kubernetes?

Question

In kubernetes I can expose services with service. This is fine.
Lets say I have 1 web instance and 10 java server instances.
I have a windows gateway I'm used to access those 10 java servers instances via the jconsole installed on it.
Obviously I do not expose all apps jmx port via kubernetes service.

What are my options here? how should I allow this external to kubernetes cluster windows gateway access to those 10 servers jmx ports? Any practices here?

daniilyar · Accepted Answer · 2018-03-26T00:41:06.630

143

Another option is to forward JMX port from K8 pod to your local PC with kubectl port-forward.

I do it like this:

1). Add following JVM options to your app:

-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.port=1099
-Dcom.sun.management.jmxremote.rmi.port=1099
-Djava.rmi.server.hostname=127.0.0.1

The critical part here is that:

The same port should be used as 'jmxremote.port' and 'jmxremote.rmi.port'. This is needed to forward one port only.
127.0.0.1 should be passed as rmi server hostname. This is needed for JMX connection to work via port-forwarding.

2). Forward the JMX port (1099) to your local PC via kubectl:

kubectl port-forward <your-app-pod> 1099

3). Open jconsole connection to your local port 1099:

jconsole 127.0.0.1:1099

This way makes it possible to debug any Java pod via JMX without having to publicly expose JMX via K8 service (which is better from security perspective).

Another option that also may be useful is to attach the Jolokia (https://jolokia.org/) agent to the Java process inside the container so it proxies the JMX over HTTP port and expose or port-forward this HTTP port to query JMX over HTTP.

edited Mar 26 '18 at 00:41

answered Oct 07 '16 at 23:44

daniilyar

2,602
3
22
25

3

This is the most straightforward explanation I've encountered and works perfectly. Thanks! – Matt Friedman Mar 08 '17 at 12:39
1

Thank you! Big help – f-z-N May 30 '18 at 14:45
8

setting `-Djava.rmi.server.hostname=127.0.0.1` did the trick for us. – jitter Jul 03 '18 at 15:29
How to add the JVM options to the app? Is there any file where those options have to be added? We're deploying a WAR file to Tomcat server in the container. – user5155835 Oct 18 '18 at 07:53
Perfect explanation. Thanks you! – drk May 16 '19 at 13:41
4

I owe you a coffee! – Dmitry Minkovsky May 22 '19 at 23:43
1

How can I connect to this jmx server from a remote host ? – bhavanak Aug 02 '19 at 11:41
@daniilyar hostname can be tricked in hosts file, but how about two diff ports (for port and rmi.port)? Any chance on correct forwarding in such case? – Sergey Narozhny May 01 '20 at 19:23
Wow. Have been searching all over for this. Now it makes sense. Thanks a lot! – Gino May 09 '20 at 00:01
This amazing answer lacks just one detail which I found in another answer https://stackoverflow.com/a/36431926/999367: "Then when I tried to use VisualVM to connect to localhost:9001, the connection was refused. JMX seems to require **port numbers on both sides to be identical**." That completed the trick for me. – Roman Sinyakov Feb 17 '21 at 18:24

score 4 · Answer 2 · answered Sep 09 '18 at 09:20

Adding to https://stackoverflow.com/a/39927197/1387184, I wanted to monitor all instances of the same pod at once, since we are hardcoding the port as 1099, it was difficult as I can do only one portforward to one pod with that port.

I used Shell Script to dynamically assign pod when running the docker

Dockerfile CMD /run.sh

run.sh

JMX_PORT=$(((RANDOM % 20)+1099))

echo "Running JMX on Port $JMX_PORT"

java ``eval echo $JAVA_OPTS`` ...

deployment.yml env: - name: JAVA_OPTS value: "-Xms256m -Xmx6144m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=$JMX_PORT -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT -Djava.rmi.server.hostname=127.0.0.1"

the eval will evaluate the JMX_PORT to bash value, each pod when starts will likely to get different pod. I

score 3 · Answer 3 · answered Feb 20 '16 at 03:01

3

We did it in following way

Add a unique label for each pod. ex: podid=asdw23443
Create a new service with selector of podid=asdw23443. Make sure in the service you expose jmx ports on pod through nodeport or loadbalancer.

If you are selecting nodeport in service, because you a doing a NAT operation you may have to give following JVM argument for each jvm you need to connect through jconsole

-Djava.rmi.server.hostname=<your-ip-address>

answered Feb 20 '16 at 03:01

Dimuthu

7,946
6
22
37

I'm not sure I get it is it that you have a podid for each instance of the pod? so if you scale it up from 10 to 20 then you have access to each and every of the pod instances without a load balancer? I want access to a specific pod instance. – Jas Feb 20 '16 at 09:13
I'm not sure I get it is it that you have a podid for each instance of the pod? so if you scale it up from 10 to 1000 then you have 1000 services? – Jas Feb 20 '16 at 09:30
Yes in this approach you have to create a service per pod. If you have 1000 pods then there should be 1000 services. – Dimuthu Feb 20 '16 at 13:43
One thing I'm struggling with: how can I set a JVM arg to the IP address running the process, when I don't know ahead of time which worker the pod will be scheduled to? – joelittlejohn Oct 06 '16 at 08:53

score 1 · Answer 4 · edited Feb 26 '16 at 12:28

1

I think one way is to add a label to your pod with a unique string \ id for example pod_name and use the expose command to create a new service with the selector of this unique id\string.

kubectl label pods <podname> podname=<podname>
kubectl expose pod <podname> --port=9010 --name=<podname>_jmx

edited Feb 26 '16 at 12:28

Mr Lister

45,515
15
108
150

answered Feb 19 '16 at 10:29

Liran Cohen

11
1

multiple app nodes how to expose jmx in kubernetes?

4 Answers4

Linked