39

I know that spring automatically expose JMX beans. I was able to access it locally using VisualVM.

However on prod how I can connect to remotely to the app using it's JMX beans? Is there a default port or should I define anything in addition?

Thanks, ray.

rayman
  • 20,786
  • 45
  • 148
  • 246

4 Answers4

58

By default JMX is automatically accessible locally, so running jconsole locally would detect all your local java apps without port exposure.

To access an app via JMX remotely you have to specify an RMI Registry port. The thing to know is that when connecting, JMX initializes on that port and then establishes a data connection back on a random high port, which is a huge problem if you have a firewall in the middle. ("Hey sysadmins, just open up everything, mkay?").

To force JMX to connect back on the same port as you've established, you have a couple of the following options. Note: you can use different ports for JMX and RMI or you can use the same port.

Option 1: Command line

-Dcom.sun.management.jmxremote.port=$JMX_REGISTRY_PORT 
-Dcom.sun.management.jmxremote.rmi.port=$RMI_SERVER_PORT

If you're using Spring Boot you can put this in your (appname).conf file that lives alongside your (appname).jar deployment.

Option 2: Tomcat/Tomee configuration

Configure a JmxRemoteLifecycleListener:

Maven Jar:

    <dependency>
        <groupId>org.apache.tomcat</groupId>
        <artifactId>tomcat-catalina-jmx-remote</artifactId>
        <version>8.5.9</version>
        <type>jar</type>
    </dependency>

Configure your server.xml:

<Listener className="org.apache.catalina.mbeans.JmxRemoteLifecycleListener"
      rmiRegistryPortPlatform="10001" rmiServerPortPlatform="10002" />

Option 3: configure programmatically

@Configuration
public class ConfigureRMI {

    @Value("${jmx.rmi.host:localhost}")
    private String rmiHost;

    @Value("${jmx.rmi.port:1099}")
    private Integer rmiPort;

    @Bean
    public RmiRegistryFactoryBean rmiRegistry() {
        final RmiRegistryFactoryBean rmiRegistryFactoryBean = new RmiRegistryFactoryBean();
        rmiRegistryFactoryBean.setPort(rmiPort);
        rmiRegistryFactoryBean.setAlwaysCreate(true);
        return rmiRegistryFactoryBean;
    }

    @Bean
    @DependsOn("rmiRegistry")
    public ConnectorServerFactoryBean connectorServerFactoryBean() throws Exception {
        final ConnectorServerFactoryBean connectorServerFactoryBean = new ConnectorServerFactoryBean();
        connectorServerFactoryBean.setObjectName("connector:name=rmi");
        connectorServerFactoryBean.setServiceUrl(String.format("service:jmx:rmi://%s:%s/jndi/rmi://%s:%s/jmxrmi", rmiHost, rmiPort, rmiHost, rmiPort));
        return connectorServerFactoryBean;
    }
}

The trick, you'll see, is the serviceUrl in which you specify both the jmx:rmi host/port and the jndi:rmi host/port. If you specify both, you won't get the random high "problem".

Edit: For JMX remoting to work, you'll need to make a decision about authenticating. It's better to do it in 3 distinct steps:

  1. basic setup with -Dcom.sun.management.jmxremote.authenticate=false then
  2. add a password file (-Dcom.sun.management.jmxremote.password.file). See here for instructions. + -Dcom.sun.management.jmxremote.ssl=false and then
  3. set up SSL.
inanutshellus
  • 9,683
  • 9
  • 53
  • 71
  • 1
    Does this approach exclude using `JAVA_OPTS` method below? – raffian Jan 12 '16 at 03:53
  • I tried this method (Option 3) with a Java service running within a Docker container, and had trouble connecting to it from outside the container. I'll have to do a small sample project to explore this idea better. – djangofan Sep 09 '18 at 23:24
  • is this `(appname).conf` file documented somewhere? – Oleksandr Shmyrko Nov 23 '20 at 14:23
  • 1
    @OleksandrSh, the documentation is here: https://docs.spring.io/spring-boot/docs/2.3.2.RELEASE/reference/html/deployment.html#deployment-script-customization-conf-file – Dima Korobskiy Jan 27 '21 at 17:04
  • Option 1 errors out (JDK 11, Spring Boot 2.3.2) with `Error: Password file not found: {skipped} jdk.internal.agent.AgentConfigurationError at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.checkPasswordFile(ConnectorBootstrap.java:572) at jdk.management.agent/sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:436) at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:447) at jdk.management.agent/jdk.internal.agent.Agent.startAgent(Agent.java:599)` – Dima Korobskiy Jan 27 '21 at 22:56
  • Well, Option 1 (`-Dcom.sun.management.jmxremote.port=$JMX_REGISTRY_PORT` + `-Dcom.sun.management.jmxremote.rmi.port=$RMI_SERVER_PORT`) is missing the required property: `-Dcom.sun.management.jmxremote.authenticate=false` or true with the setup from @Popyeye answer. It's just not working without this in OpenJDK 11. That's the root cause for the `Error: Password file not found`. – Dima Korobskiy Feb 09 '21 at 17:57
  • @DKroot Java 11 is no different than Java 7+ in this regard. The authenticate flag is true by default and you're supposed to configure a `jmxremote.password` file. Setting the flag to false is a security vulnerability so don't do it unless you're on a non-prod machine. – inanutshellus Feb 10 '21 at 20:22
  • Yes, I went with @Popeye's setup. Overall, JMX configuration was incredibly hairy. I think it's better to do it in 3 distinct steps: 1) basic setup with `-Dcom.sun.management.jmxremote.authenticate=false` then 2) add a password file and `-Dcom.sun.management.jmxremote.ssl=false` and then 3) set up SSL – Dima Korobskiy Feb 11 '21 at 14:27
  • @DKroot That's reasonable advice. Feel free to update the ticket to say so if you like, or I can if you'd prefer not to. – inanutshellus Feb 11 '21 at 16:50
  • @inanutshellus Sure. Which ticket would I need to update? – Dima Korobskiy Feb 12 '21 at 18:37
  • @DKroot Ha! I said "ticket" out of habit. I meant add your advice to the SO Answer above. – inanutshellus Feb 12 '21 at 19:06
  • 1
    @inanutshellus Done. OMG: you can edit somebody else's answer! It's the first time I did it on SO. My own answers: sure. Somebody else's: blew my mind. :) – Dima Korobskiy Feb 14 '21 at 22:29
40

Add the following JVM Properties in "$JAVA_OPTS" (in your application):

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=<PORT_NUMBER> -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=<HOST'S_IP>

In the Jconsole/Visual VM use the following to connect:

service:jmx:rmi:///jndi/rmi://<HOST'S_IP>:<PORT_NUMBER>/jmxrmi

It doesn't enable security, but will help you to connect to the remote server.

kryger
  • 12,906
  • 8
  • 44
  • 65
Arnab Biswas
  • 4,495
  • 3
  • 42
  • 60
  • This did the trick for me ... thank you. It did not work until I added the -Djava.rmi.server.hostname. I was able to connect via jconsole like this: ip:port – em_bo May 16 '18 at 23:06
  • 1
    These params have to be used via the `java` command line. Setting them in JAVA_OPTS didn't work for me for Spring Boot 2.3.2. Here is a longer blog post: https://medium.com/@cl4r1ty/docker-spring-boot-and-java-opts-ba381c818fa2. (Docker does not matter, `JAVA_OPTS="..." java -jar` doesn't work either.) – Dima Korobskiy Jan 27 '21 at 23:08
6

A tested approach on Java 1.8.0_71 and Spring Boot(1.3.3.RELEASE). Append below parameters to JVM arguments for monitored JVM.

-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=12348 -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.rmi.port=12349 -Dcom.sun.management.jmxremote.password.file=/somewhere/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/somewhere/jmx/jmxremote.access

The com.sun.management.jmxremote.port is used to define the fixed RMI registry port, and the com.sun.management.jmxremote.rmi.port is used to instruct JVM to use fixed RMI port, but NOT use random one.

By setting this, I am able to connect JVM client from remote host to the monitored JVM via a firewall just opening 12348 and 12349 port.

I tested using java -jar cmdline-jmxclient-0.10.3.jar user:pwd hostip:12348 on a remote machine, which generates below output(shortened just for demonstration).

java.lang:type=Runtime
java.lang:name=PS Scavenge,type=GarbageCollector
Tomcat:J2EEApplication=none,J2EEServer=none,WebModule=//localhost/,j2eeType=Filter,name=requestContextFilter
java.nio:name=mapped,type=BufferPool
Tomcat:host=localhost,type=Host
java.lang:name=Compressed Class Space,type=MemoryPool
.......

The jar is downloaded from Here.

Popeye
  • 2,002
  • 20
  • 14
  • I tested this approach successfully in OpenJDK 11 with 2 improvements: 1. Using the same port for JMX and RMI. 2. Omitting `-Dcom.sun.management.jmxremote`: this is no longer needed. – Dima Korobskiy Feb 09 '21 at 19:26
0

Another alternative

Reference for jmxremote.password and jmxremote.access files

import java.util.HashMap;
import java.util.Map;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.DependsOn;
import org.springframework.jmx.support.ConnectorServerFactoryBean;
import org.springframework.remoting.rmi.RmiRegistryFactoryBean;

@Configuration
public class ConfigureRMI {

    @Value("${jmx.rmi.password.file:/tmp/jmxremote.password}")
    private String passwordFile;

    @Value("${jmx.rmi.access.file:/tmp/jmxremote.access}")
    private String accessFile;

    @Value("${jmx.rmi.port:19999}")
    private Integer rmiPort;

    @Bean
    public RmiRegistryFactoryBean rmiRegistry() {
        final RmiRegistryFactoryBean rmiRegistryFactoryBean = new RmiRegistryFactoryBean();
        rmiRegistryFactoryBean.setPort(rmiPort);
        rmiRegistryFactoryBean.setAlwaysCreate(true);
        return rmiRegistryFactoryBean;
    }

    @Bean
    @DependsOn("rmiRegistry")
    public ConnectorServerFactoryBean connectorServerFactoryBean() throws Exception {
        final ConnectorServerFactoryBean connectorServerFactoryBean = new ConnectorServerFactoryBean();
        connectorServerFactoryBean.setObjectName("connector:name=rmi");
        Map<String, Object> properties = new HashMap<>();
        properties.put("jmx.remote.x.password.file", passwordFile);
        properties.put("jmx.remote.x.access.file", accessFile);
        connectorServerFactoryBean.setEnvironmentMap(properties);
        connectorServerFactoryBean.setServiceUrl(String.format("service:jmx:rmi:///jndi/rmi://:%s/jmxrmi", rmiPort));
        return connectorServerFactoryBean;
    }
}