5

Sorry if this is a repeat of a question asked previously, but I have tried out the solutions provided here and elsewhere and none of them seem to work for me.

Scenario: I need to profile a production Tomcat 7 server that lives behind a firewall. I only have SSH access to this server. The profiling tool of choice is Visualvm and these servers do not have a GUI installed. Both the server and the machine I am trying to connect from run Ubuntu 10.04 server LTS.

Possible solution: Using visualvm over SSH tunnels seem to be possible and the link given below gives a pretty detailed guide to doing this:

Tomcat+VisualVM+SSH guide

Also I have added a jmxremote.access file to the /conf dir in Tomcat with the following two lines: 

monitorRole readonly
controlRole readwrite

Problem: I followed the steps provided in the article and everything seems to be fine. I can connect to the jmx ports in the server locally (from the server) using jmxterm and the connection string given in the guide.

I configured local tomcat as per the guide as a test and Visualvm had no problem connecting to this tomcat instance using the connection string.

However, VisualVM will not connect to the remote server over SSH tunnel. I tried JConsole as it it simpler and has better logging facilities and I get the following sequence of messages:

FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi://localhost:12009/jndi/rmi://localhost:12008/jmxrmi]  connecting...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi://localhost:12009/jndi/rmi://localhost:12008/jmxrmi] finding stub...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi://localhost:12009/jndi/rmi://localhost:12008/jmxrmi] connecting stub...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi://localhost:12009/jndi/rmi://localhost:12008/jmxrmi] getting connection...
FINER: [javax.management.remote.rmi.RMIConnector: jmxServiceURL=service:jmx:rmi://localhost:12009/jndi/rmi://localhost:12008/jmxrmi] failed to connect: java.rmi.NoSuchObjectException: no such object in table

Any ideas as to what I've done wrong here? Any help is greatly appreciated!

Thank you.

ChamaraG
  • 95
  • 1
  • 7

2 Answers2

2

I think the issue you are having, which the referenced article does not really address, is the way that jmx-rmi, or more specifically, RMI itself works. Your Tomcat server setup like this:

  1. Created an RMI remotable that is configured to connect to localhost:12009.
  2. Created an RMI registry on localhost with a listening port of localhost:12008.

When you run locally, you lookup the stub in the RMI registry, and when you invoke against the stub to issue a jmx call, it connects to localhost:12009, no problem.

When you run remotely through SSH, I am assuming you have tunneled from your remote to the tomcat server on port 12008, so when you issue your request, your remote's localhost:12008 is tunneled to tomcatServer:12008 since the log output seems to indicate that you found the stub ok.

Next, your remote executes against the stub which in turn attempts to connect to localhost:12009 and fails.

My guess is that you need to tunnel your remote's localhost:12008 and localhost:12009 to be redirected to tomcatServer:12008 and tomcatServer:12009.

Another way of getting around this is to ditch the RMI jmx protocol and use something like Jolokia (HTTP), JMX-WS (HTTP WebService) or JMXMP (Socket). This will make your SSH tunelling life a lot easier. If you want to try JMXMP, I created a github project called OpenDMK which has mavenized builds. You will want the jmx-optional package.

Nicholas
  • 15,916
  • 4
  • 42
  • 66
  • Nicholas, thank you for the answer. I do have both localhost:12008 and localhost:12009 tunneled to the corresponding ports on the tomcat server. It is just not working. I took your suggestion to try JMXMP and it works magnificently. Thank you. – ChamaraG Sep 08 '12 at 02:35
  • You may use standard RMI and SSH tunneling, see https://stackoverflow.com/questions/1609961/visualvm-over-ssh/31185858#31185858 – Thies May 17 '18 at 15:31
0

Thanks for your question! -- I had the same basic need. Sorry, I'm not actually adding anything new here but thought I would mention that the reference you provided: https://www.liferay.com/web/thiago.moreira/blog/-/blogs/how-to-monitor-liferay-tomcat-remotely-through-firewalls-using-visualvm just worked right away for me (with just very minor adjustments only related with more recent versions -- Tomcat 7, and VisualVM 1.3.8). In my case Tomcat is running on CentOS 6.6, and the monitoring machine is a Mac (perhaps this could help somebody else out there). Thanks again.

carueda
  • 182
  • 1
  • 6