6

I've got a Java8 application running on RHEL 6.10. This application registers a shutdown handler via the usual method:

Thread shutdownThread = new Thread(()=>{
   Logger.info("Got shutdown signal");
   // Do cleanup
});
Runtime.getRuntime().addShutdownHook(shutdownThread);

This application is kicked off by a Jenkins build (with the BUILD_ID env var set to dontkillme). The application initializes successfully, but then after ~30 seconds the shutdown hook is called and the application terminates. I'm trying to figure out who is shutting me down and why. I've monitored top and it doesn't appear that memory is an issue while it's running, so I don't think the OOM killer is the culprit. I've also looked at /var/log/dmesg and /var/log/messages and don't see anything relevant there either. I don't think Jenkins would be killing me, both since I set BUILD_ID and also because the application dies while the "parent" Jenkins job is still running.

What other methods / tools can I use to see what's happening? Note that my environment is very locked down, so it would be difficult to download and run something from the internet, hopefully there's something in a standard RHEL6 install I could use.

redsoxfantom
  • 918
  • 1
  • 14
  • 29
  • 3
    OOM killer wouldn't call a shutdown hook either. Something is sending a sigkill. – Michael Mar 19 '20 at 17:27
  • yeah, figured that. Just thought i'd add it – redsoxfantom Mar 19 '20 at 17:29
  • 1
    _I don't think Jenkins would be killing me_ Seems fairly easy to eliminate Jenkins by running your app as stand alone, i.e. launch from command line and see if it still gets shut down. – Abra Mar 19 '20 at 17:31
  • It runs standalone successfully. But I thought I was configuring things so Jenkins wouldn't kill me (for example, there are other apps the job starts and they outlive the parent job). Unless anyone knows other ways to confirm Jenkins isnt the bad guy – redsoxfantom Mar 19 '20 at 17:33

1 Answers1

0

The answer turned out to be unique to the application in question. The application uses the zookeeper-3.5.5 library to connect to a Zookeeper instance. This client library has a runtime dependency on the zookeeper-jute jar, and when that jar was not present in the executing directory, this issue presented itself.

You're probably wondering why the application silently shut itself down and didn't throw a ClassNotFoundException which would have helped me debug this. Great question! I have no idea.

redsoxfantom
  • 918
  • 1
  • 14
  • 29
  • Applications don't just shut themselves down in Java. There are only a a few ways (on less than one hand's worth of fingers) that a JVM just stops. Your application might only have one thread and the ClassNotFoundException is propagating back to the main method and the script you're using is swallowing I/O, or you're catching the exception and calling `System#exit(int)`, neither of which you should be doing in the first place. Either way, stepping through with a debugger sounds like what you want to be doing instead of playing the guessing game. Your code still has the problem; library or not – searchengine27 Mar 23 '20 at 16:50