Allow restarting Java application with JMX monitoring enabled immediately

Question

I have a Java application with JMX monitoring enabled like this:

-Dcom.sun.management.jmxremote.port=9999 \
// some other properties omitted

But when I try to restart the application, sometime I got an error says the JMX port number is already in use. This is not acceptable.

So I want to set the SO_REUSEADDR to true for the underlying socket to avoid this error but found no related JMX properties.

Any idea?

I must be my application. When I stop the application, I think the socket bound to this port goes to TIME_WAIT state for 2MSL to close actually. So I want to make this port reusable. — George, May 08 '14 at 02:46
SO_REUSEADDR does not work that way. It allows sockets to listen to specific IP addresses and ignore others. Either the same application is running twice or there is another application grabbing this port. — BevynQ, May 08 '14 at 02:57
@BevynQ, if I stop the application and do NOT start it immediately and later I try to start the application, I works. So I think it's the socket TIME_WAIT mechanism disallow — George, May 08 '14 at 03:06
@George You say you "think" it is. Is it or isn't it? What process does netstat say is using that port, and what state is it in? — Neil Coffey, Apr 24 '15 at 13:18
@BevynQ i doublechecked and I believe OP is correct in what SO_REUSEADDR does. — Fox, Apr 24 '15 at 13:27
@NeilCoffey netstat does not name a process for "TIME_WAIT" if the process is already finished, e.g. `tcp 0 0 localhost:57525 localhost:3100 TIME_WAIT -`. There are also some questions about SO_REUSEADDR in general (like http://stackoverflow.com/q/3229860/602119) which also mention TIME_WAIT in the answers. — Boris, Apr 24 '15 at 13:31
The jmx server-port is created by the JVM, but a fairly easy way to create the server-socket myself would be a good workaround or even a solution. — Boris, Apr 24 '15 at 13:38
@Boris Ah sorry OK, if it doesn't, my bad -- I thought it did still list it even if there was no process attached (maybe this varies?) — Neil Coffey, Apr 25 '15 at 14:42

score 7 · Answer 1 · answered Apr 24 '15 at 13:37

7

I am afraid you can't do that from command line.

You will need to create a RMIServerSocketFactory, which produces ServerSockets with the desired option (SO_REUSEADDR).

Docs here: http://docs.oracle.com/javase/8/docs/technotes/guides/rmi/socketfactory/

Someone else solving the same problem: https://svn.apache.org/viewvc?view=revision&revision=r1596579

answered Apr 24 '15 at 13:37

Fox

2,348
19
19

Thanks. I'm afraid that might indeed be the only solution. Putting it all together it's much more verbose than I'd hoped, but still better then nothing. – Boris Apr 24 '15 at 18:48

score 1 · Answer 2 · answered Apr 30 '15 at 13:28

I had the same issue. It was the first instance of my application (which I had stopped) which was still subscribed to this port, so the new instance could not start. In my case it didn't have to do with the socket TIME_WAIT mechanism but rather with the fact that after calling stop(), it took some time until all running threads ended gracefully. What worked in my case was unregistering the bean right before stopping the application so that the socket is free.

private void unregisterBeanForName(String name) {
        try {

            JMXServiceURL jmxServiceURL = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://127.0.0.1:9999/jmxrmi");
            JMXConnector cc = JMXConnectorFactory.connect(jmxServiceURL);
            MBeanServerConnection mbsc = cc.getMBeanServerConnection();
//This information is available in jconsole
            ObjectName serviceConfigName = new ObjectName("YourObjectName");
            mbsc.unregisterMBean(serviceConfigName);
// Close JMX connector
            cc.close();
        } catch (Exception e) {
            System.out.println("Exception occurred: " + e.toString());
            e.printStackTrace();
        }
    }

Sergey Alaev · Answer 3 · 2015-04-28T20:43:05.667

Yes, you should create JMX connector programmatically. As easier workaround, you can select another port for JMX in runtime if default port is busy by just killed process. OR just attempt opening your port again and again until it succeeds.

Here is code snippet i use to open JConsole-compatible JMX connector. In Scala, sorry, but you should be able to adapt it easily enough

def startJmx(port: Int): Unit = {
if (port < 1) {
  return
}

val log = LoggerFactory.getLogger(getClass)

log.info("Starting JMX server connector on port {}", port)

val registry = LocateRegistry.createRegistry(port)

val server = ManagementFactory.getPlatformMBeanServer()

val url = new JMXServiceURL(s"service:jmx:rmi:///jndi/rmi://localhost:$port/jmxrmi")

val connectorServer = JMXConnectorServerFactory.newJMXConnectorServer(url, Collections.emptyMap(), server)

 val thread = new Thread {
   override def run = try {
     connectorServer.start()
   } catch { 
     case e: Exception => log.error("Unable to start JMX connector", e)
   }
 }
 thread.setDaemon(true)
 thread.setName("JMX connector Thread")
 thread.start()
}

score 0 · Answer 4 · answered May 01 '15 at 13:14

0

This could be one workaround: On the remote server you can have two ports: 9999 and 9998 forwarded to 9999.

When restarting your application alternate a boolean each time to decide to connect to 9999 or 9998.

answered May 01 '15 at 13:14

pjsofts

170
4
18

score -1 · Answer 5 · answered Apr 27 '15 at 05:56

-1

Add a shutdown hook on your application that will kill jmx.

// kill process with port 9999    
fuser -k 9999/tcp

answered Apr 27 '15 at 05:56

jjcosare

1,463
9
9

This would be platform-dependent (which would be OK in my case) but more importantly it doesn't work. Killing the process via fuser does not seem to change the behaviour regarding SO_REUSEADDR. I tried with a simple class and still got the BindException – Boris Apr 27 '15 at 12:25

Allow restarting Java application with JMX monitoring enabled immediately

5 Answers5