Fatal errors from openjdk when running fresh jar files

Question

I want to implement an ApplicationChangeMonitor which monitors the filesystem for changes in the currently executed jar file. When a change is detected the application should restart. I'm using a WatchService to detect the changes.

The setup:

• Development in (Windows) Eclipse with a workspace on a samba share (Linux system)
• The jar file is generated by Eclipse maven (m2e) on that samba share
• The jar file is executed from shell on the Linux system (using openjdk)

So everytime a new jar file is created, the running application should be restarted on the Linux system. First I tried to make the application restart itself, but most of the times I ran into fatal errors from the JVM. Then I chose a simpler approach: I just made the application end itself after a change is detected and implemented the restart mechanism with bash:

while true ; do java -jar application.jar ; done

The weird thing is, I still get fatal errors once or twice after an application change. Example:

• java -jar application.jar <-- initial start, application is running
• New jar file created
• java -jar application.jar <-- fatal error
• java -jar application.jar <-- fatal error
• java -jar application.jar <-- application starts
• New jar file created
• java -jar application.jar <-- fatal error
• java -jar application.jar <-- application starts

The output:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f46d5e2416d, pid=28351, tid=139942266005248
#
# JRE version: 7.0_25-b30
# Java VM: OpenJDK 64-Bit Server VM (23.7-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libzip.so+0x516d]  Java_java_util_zip_ZipFile_getZipMessage+0x114d
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/workspace/.../target/hs_err_pid28351.log
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   http://icedtea.classpath.org/bugzilla
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

OpenJDK creates dump files and I guess the relevant part is the stack trace leading to this fatal error:

 - Stack: [0x00007fbc9398f000,0x00007fbc93a90000],  sp=0x00007fbc93a8bd90,  free space=1011k
 - Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
 - C  [libzip.so+0x516d]  Java_java_util_zip_ZipFile_getZipMessage+0x114d
 - C  [libzip.so+0x5eb0]  ZIP_GetEntry+0xd0
 - C  [libzip.so+0x3af3]  Java_java_util_zip_ZipFile_getEntry+0xb3
 - j  java.util.zip.ZipFile.getEntry(J[BZ)J+0
 - j  java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry;+38
 - j  java.util.jar.JarFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry;+2
 - j  java.util.jar.JarFile.getJarEntry(Ljava/lang/String;)Ljava/util/jar/JarEntry;+2
 - j  sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;+48
 - j  sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource;+53
 - j  java.net.URLClassLoader$1.run()Ljava/lang/Class;+26
 - j  java.net.URLClassLoader$1.run()Ljava/lang/Object;+1
 - ...

Now, does anyone have any idea why I get these fatal errors? I thought maybe it's because the jar file hasn't been written completely (that would explain why the problem comes from Java_java_util_zip_ZipFile_getZipMessage). But that's simply not the case because the md5sum of the jar stays the same after executions resulting in fatal errors and working executions.

while true; do md5sum application.jar ; java -jar application.jar ; done

Answer 1

This is because you are getting notified for new file, while that file is being written to disk. This is bad thing with WatchService, it will notify you as soon as new file gets created but not yet written completely to disk.

When new jar file is being written to disk, the jar file is locked by process, which is writing that jar file to disk. You can't access file till file creator process has not unlocked file.

The solution for this: You have to try to open file, if file gets opened then file has been written to disk completely. If you fail to open file, then wait for some time(or dont wait, try next), and try next to open file.

To unlock file, implement something like this:

public void unlockFile(String jarFileName){
    FileInputStream fis = null;
    while(true){
        try{
            // try to open file
            fis = new FileInputStream(jarFileName);
            // you succeed to open file
            // return
            // file will be closed in finally block, as it will always executed
            return;
        }catch(Exception e){
            // file is still locked
            // you may sleep for sometime to let other process finish with file and
            // file gets unlocked

            // if you dont have problem with this process utilizing CPU, dont sleep!
            try{
                Thread.sleep(100);
            }catch(InterruptedException ie){
            }
        }finally{
            if(fis != null){
                try{
                    fis.close();
                }catch(Exception e){
                }
            }
        }
    }

To your problem,

You told: "I just made the application end itself after a change is detected and implemented the restart mechanism with bash"

So, before you end java process, unlock file as suggested by me in above method. I am sure errors will go away. Just try and let me know results.

Something like this:

void shutDownMethod(){
    // get file name from watcher, below line will depend on your logic and code.
    String jarFileName = watcherThread.getNewNotifiedFile();
    // unlock new jar file.
    unlockFile(jarFileName);
    // shutdown JVM
    System.exit(0);
    // bash will restart JVM
}