Guess method line number from JVM fatal error log

Question

I have a fatal error log without a core-dump and need to identify the cause. This is the stack found in the .log file:

# Problematic frame:
# C  [libc.so.6+0x7b4bb]  memcpy+0x15b

{...}

Stack: [0x00002ac8c4d2c000,0x00002ac8c4e2d000],  sp=0x00002ac8c4e28ef8,  free space=1011k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libc.so.6+0x7b4bb]  memcpy+0x15b
C  [libzip.so+0x50b0]  ZIP_GetEntry+0xd0
C  [libzip.so+0x3eed]  Java_java_util_zip_ZipFile_getEntry+0xad
J 28  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00002ac8acf404ee [0x00002ac8acf40420+0xce]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 28  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00002ac8acf40478 [0x00002ac8acf40420+0x58]
J 33 C2 java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (86 bytes) @ 0x00002ac8acf45548 [0x00002ac8acf45480+0xc8]
J 58 C2 java.util.jar.JarFile.getJarEntry(Ljava/lang/String;)Ljava/util/jar/JarEntry; (9 bytes) @ 0x00002ac8acf4e828 [0x00002ac8acf4e7e0+0x48]
J 44 C2 sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (91 bytes) @ 0x00002ac8acf47168 [0x00002ac8acf47100+0x68]
J 34 C2 sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (74 bytes) @ 0x00002ac8acf41f70 [0x00002ac8acf41f00+0x70]
j  java.net.URLClassLoader$1.run()Ljava/lang/Class;+26
j  java.net.URLClassLoader$1.run()Ljava/lang/Object;+1
v  ~StubRoutines::call_stub
j  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object;+0
j  java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class;+13
j  java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+70
j  sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+36
j  java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class;+3
v  ~StubRoutines::call_stub
j  com.smackdapp.SmackHandler.handleRequest(Ljava/lang/String;Lorg/eclipse/jetty/server/Request;Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+72

As I have read in another StackOverflow answers, this can be a common error while replacing any .jar that the jvm needs to load (or a hardware memory error).

What I'm trying to guess is the .jar file that caused this fatal error. Is there any way to identify the line number in my source code with the information in the fatal error log?

I was hoping that this "V+72" at the end of the line has something to do, but can't figure out.

I've also read oracle's doc about fatal error log files http://www.oracle.com/technetwork/java/javase/felog-138657.html but didn't help. — Turin, Mar 20 '15 at 17:56
"V" simply means it's a void method. I suspect the +72 is the bytecode offset into the method, but I wouldn't swear to it. — Hot Licks, Mar 20 '15 at 18:01
Someone downvoted this - maybe because he thought that this could be considered as "off-topic as asking for debugging help" or so. But in fact, it's an interesting and relevant question for many developers, so 1 up from me. — Marco13, Mar 20 '15 at 18:02
Run your program again with -Xint and see if it still fails. java -Xint myProgram. Report back — Amir Afghani, Mar 20 '15 at 18:14
Thank you @HotLicks. I'm trying to match with the bycode reported by IntelliJ, but there are a lot of comments. Does not seems accurate. — Turin, Mar 20 '15 at 18:26
@AmirAfghani the problem is that its a production problem, not easy to reproduce. In fact, this error happened in two java processes, with 2 hours delay. Also, if I could reproduce it, the first to try will be enable core dumps I guess. — Turin, Mar 20 '15 at 18:28
I'm not sure what you mean by "the bytecode reported by IntelliJ". Use *javap* to dump the class and see what's at offset 72 in that method. — Hot Licks, Mar 20 '15 at 18:30
@HotLicks Guess he's refering to something like this plugin https://plugins.jetbrains.com/plugin/5918?pr= - The line numbers should match, but `javap` should be used to counter-check this. — Marco13, Mar 20 '15 at 18:50

score 5 · Accepted Answer · edited May 23 '17 at 10:29

This may not help to solve the problem, eventually, but it answers a part of the question:

As already guessed by Hot Licks in his comment, the +72 just is the bytecode offset. I tested this with a small program:

^{(otherwise unrelated, just had it quickly available here from another question)}

class TestNativeArray3D
{
    public static void main(String args[])
    {
        System.loadLibrary("TestNativeArray3D");
        int terrain[][][] = genTerrain(123, 8, 6);
    }
    private native static int[][][] genTerrain(int seed, int x, int y);
}

The native genTerrain function blatantly creates an arbitrary error, by calling

jclass errorClass = env->FindClass("XXX");
env->NewObjectArray(10, errorClass, NULL);

causing a core dump.

The stack looks as follows:

Stack: [0x0000000002500000,0x0000000002600000],  sp=0x00000000025ff4e0,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0x14ebb6]
C  [TestNativeArray3D.dll+0x397d]  JNIEnv_::NewObjectArray+0x4d
C  [TestNativeArray3D.dll+0x3ae8]  Java_TestNativeArray3D_genTerrain+0x98
C  0x0000000002715534

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  TestNativeArray3D.genTerrain(III)[[[I+0
j  TestNativeArray3D.main([Ljava/lang/String;)V+11
v  ~StubRoutines::call_stub

Here, the line that analogously contains the (possible) offset is

j TestNativeArray3D.main([Ljava/lang/String;)V+11

When decompiling the class file with

javap -c -l TestNativeArray3D

(note the -l (small "L") to obtain the line numbers!) the output is

class TestNativeArray3D {
  TestNativeArray3D();
    Code:
       0: aload_0
       1: invokespecial #1                  // Method java/lang/Object."<init>":()V
       4: return
    LineNumberTable:
      line 3: 0

  public static void main(java.lang.String[]);
    Code:
       0: ldc           #2                  // String TestNativeArray3D
       2: invokestatic  #3                  // Method java/lang/System.loadLibrary:(Ljava/lang/String;)V
       5: bipush        123
       7: bipush        8
       9: bipush        6
      11: invokestatic  #4                  // Method genTerrain:(III)[[[I
      14: astore_1
      15: return
    LineNumberTable:
      line 7: 0
      line 8: 5
      line 9: 15
}

and indeed, the native call happens at offset 11. (And yes, I counter-checked this by adding some further code before this call: The offset in the core dump changes accordingly).

The "LineNumberTable" will allow you to map between bytecode offsets and lines. The table in the example means that

source code line 7 corresponds to bytecode offset 0,
source code line 8 corresponds to bytecode offset 5,
source code line 9 corresponds to bytecode offset 15

so the critical instruction here is related to code line 8. (This was obvious, in this case, because this was directly the invokestatic call. I just wanted to point out that you can look up the LineNumberTable for the mapping of the bytecode offset to the actual line number in the source code)

Unfortunately, this will hardly help you to really solve the error, because this one was obviously caused some levels deeper, in the native code of libc.so, during some memcpy - which obviously received invalid pointers from the libzip.so...

Thank you Marco, and @HotLicks for the initial tip about using javap. As you pointed, this does not solve the error, but at least give me a clue about what jar was trying to by loaded by the jvm. — Turin, Mar 20 '15 at 19:31

Guess method line number from JVM fatal error log

1 Answers1

Linked

Related