15

I've seen this question and this article on how to debug a native Java crash. The article is with respect to Windows. What are the equivalent debugging aids on Linux?

Note: All I have is this crash log from a user in the field. I do not have access to the machine on which the crash occurred.

Update: I am pretty sure the crash is due to JNI code we have. I never meant to imply that it was the JVM itself that was faulty.

Per request, here is the crash dump (or as much of it as will fit in the 30K stackoverflow limit):

#
# An unexpected error has been detected by Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x06300e76, pid=9983, tid=4106996592
#
# Java VM: Java HotSpot(TM) Client VM (1.6.0_03-b05 mixed mode, sharing)
# Problematic frame:
# V  [libjvm.so+0x300e76]
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x0922e000):  VMThread [id=9985]

siginfo:si_signo=11, si_errno=0, si_code=1, si_addr=0x00000008

Registers:
EAX=0x00000008, EBX=0x88a829b3, ECX=0x88a829b0, EDX=0xa7d6c1dc
ESP=0xf4cbba5c, EBP=0xf4cbba68, ESI=0xa7d6d1d8, EDI=0x00000404
EIP=0x06300e76, CR2=0x00000008, EFLAGS=0x00010202

Top of Stack: (sp=0xf4cbba5c)
0xf4cbba5c:   a7d6c1c8 0920cc30 aa0de5c0 f4cbba98
0xf4cbba6c:   063517d7 cf8f2a20 a7d6c1c8 0920cc30
0xf4cbba7c:   0920cc30 00000000 00000000 6d224c40
0xf4cbba8c:   00000001 f4cbbbb0 0920b440 f4cbbab8
0xf4cbba9c:   061dd4df 0920cc30 f4cbbb10 f4cbbac8
0xf4cbbaac:   0633cb7e 0643b5b8 f4492968 f4cbbad8
0xf4cbbabc:   061dcd68 f4cbbaf0 0920cc30 f4cbbaf8
0xf4cbbacc:   061df31e f4cbbb10 d4cbcc2c f4cbbb08 

Instructions: (pc=0x06300e76)
0x06300e66:   82 39 f2 73 34 90 8d 74 26 00 8b 02 85 c0 74 22
0x06300e76:   8b 18 80 3d 45 10 42 06 00 74 0c 89 d8 31 c9 83 

Stack: [0xf4c3c000,0xf4cbd000),  sp=0xf4cbba5c,  free space=510k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x300e76]
V  [libjvm.so+0x3517d7]
V  [libjvm.so+0x1dd4df]
V  [libjvm.so+0x1dcd68]
V  [libjvm.so+0x1dc3cc]
V  [libjvm.so+0x1d4c52]
V  [libjvm.so+0x1d32cc]
V  [libjvm.so+0x1d4229]
V  [libjvm.so+0x1dc82a]
V  [libjvm.so+0x1d1d34]
V  [libjvm.so+0x186125]
V  [libjvm.so+0x1d20bc]
V  [libjvm.so+0x3b2cbe]
V  [libjvm.so+0x3c5037]
V  [libjvm.so+0x3c46bc]
V  [libjvm.so+0x3c488a]
V  [libjvm.so+0x3c446f]
V  [libjvm.so+0x30b719]
C  [libpthread.so.0+0x5cb2]

VM_Operation (0xf2b60728): generation collection for allocation, mode: safepoint, requested by thread 0x09449c00


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x092afc00 JavaThread "RawImageCache" daemon [_thread_blocked, id=10026]
  0xf37d1000 JavaThread "TimerQueue" daemon [_thread_blocked, id=10022]
  0x09410000 JavaThread "SunTileScheduler0Standard7" daemon [_thread_blocked, id=10021]
  0x0940f000 JavaThread "SunTileScheduler0Standard6" daemon [_thread_blocked, id=10020]
  0x0946fc00 JavaThread "SunTileScheduler0Standard5" daemon [_thread_blocked, id=10019]
  0x0946e800 JavaThread "SunTileScheduler0Standard4" daemon [_thread_blocked, id=10018]
  0x0946d400 JavaThread "SunTileScheduler0Standard3" daemon [_thread_blocked, id=10017]
  0x0946c000 JavaThread "SunTileScheduler0Standard2" daemon [_thread_blocked, id=10016]
  0x0946ac00 JavaThread "SunTileScheduler0Standard1" daemon [_thread_blocked, id=10015]
  0x0946a000 JavaThread "SunTileScheduler0Standard0" daemon [_thread_blocked, id=10014]
  0x0944a800 JavaThread "Image List Poller" [_thread_blocked, id=10012]
  0x09449c00 JavaThread "Image Task Queue" [_thread_blocked, id=10011]
  0xf37e3c00 JavaThread "Laf-Widget fade tracker" [_thread_blocked, id=10010]
  0x094abc00 JavaThread "FileCacheMonitor" daemon [_thread_blocked, id=10009]
  0xf37e3800 JavaThread "DestroyJavaVM" [_thread_blocked, id=9984]
  0xf37ee400 JavaThread "Thread-6" daemon [_thread_blocked, id=10006]
  0xf3a7c800 JavaThread "DirectoryMonitor.MonitorThread" daemon [_thread_blocked, id=10005]
  0xf3a73800 JavaThread "AWT Watchdog" daemon [_thread_blocked, id=10004]
  0xf3adb800 JavaThread "TileReaper" daemon [_thread_blocked, id=10003]
  0x093c3c00 JavaThread "process reaper" daemon [_thread_in_native, id=10001]
  0x093ac800 JavaThread "Timer-0" daemon [_thread_blocked, id=9999]
  0x093a8c00 JavaThread "AWT-EventQueue-0" [_thread_blocked, id=9997]
  0x093a8000 JavaThread "AWT-Shutdown" [_thread_blocked, id=9996]
  0x09378c00 JavaThread "AWT-XAWT" daemon [_thread_blocked, id=9994]
  0x09368400 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=9993]
  0x09350000 JavaThread "Thread-1" daemon [_thread_blocked, id=9992]
  0x0923b400 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=9990]
  0x09239c00 JavaThread "CompilerThread0" daemon [_thread_blocked, id=9989]
  0x09238800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=9988]
  0x09230800 JavaThread "Finalizer" daemon [_thread_blocked, id=9987]
  0x0922f400 JavaThread "Reference Handler" daemon [_thread_blocked, id=9986]

Other Threads:
=>0x0922e000 VMThread [id=9985]
  0x09245000 WatcherThread [id=9991]

VM state:at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x09205178/0x092051a0] Threads_lock - owner thread: 0x0922e000
[0x09205638/0x09205650] Heap_lock - owner thread: 0x09449c00

Heap
 def new generation   total 83968K, used 9280K [0x55600000, 0x5b110000, 0x5ec40000)
  eden space 74688K,   0% used [0x55600000, 0x55600000, 0x59ef0000)
  from space 9280K, 100% used [0x5a800000, 0x5b110000, 0x5b110000)
  to   space 9280K,   0% used [0x59ef0000, 0x59ef0000, 0x5a800000)
 tenured generation   total 1233640K, used 1233529K [0x5ec40000, 0xaa0fa000, 0xcf800000)
   the space 1233640K,  99% used [0x5ec40000, 0xaa0de5c0, 0x8b4af400, 0xaa0fa000)
 compacting perm gen  total 13312K, used 13175K [0xcf800000, 0xd0500000, 0xd3800000)
   the space 13312K,  98% used [0xcf800000, 0xd04ddd70, 0xd04dde00, 0xd0500000)
    ro space 8192K,  69% used [0xd3800000, 0xd3d8f608, 0xd3d8f800, 0xd4000000)
    rw space 12288K,  57% used [0xd4000000, 0xd46eee98, 0xd46ef000, 0xd4c00000)

Dynamic libraries:

[ snip ]

VM Arguments:
jvm_args: -Dinstall4j.jvmDir=/home/berbmit/bin/LightZone/jre -Dinstall4j.appDir=/home/berbmit/bin/LightZone -Dexe4j.moduleName=/home/berbmit/bin/LightZone/LightZone -Dcom.lightcrafts.licensetype=ESD -Xmx2000000k
java_command: com.install4j.runtime.Launcher launch com.lightcrafts.platform.linux.LinuxLauncher true false /home/berbmit/bin/LightZone/LightZone.log /home/berbmit/bin/LightZone/LightZone.log false true false  true true -1 -1  20 20 Arial 0,0,0 8 500  20 40 Arial 0,0,0 8 500 -1
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=/home/berbmit/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
USERNAME=berbmit
LD_LIBRARY_PATH=/home/berbmit/bin/LightZone/jre/lib/i386/client:/home/berbmit/bin/LightZone/jre/lib/i386:/home/berbmit/bin/LightZone/jre/../lib/i386:/home/berbmit/bin/LightZone/.:
SHELL=/bin/bash
DISPLAY=:0.0

Signal Handlers:
SIGSEGV: [libjvm.so+0x3b29c0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGBUS: [libjvm.so+0x3b29c0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGFPE: [libjvm.so+0x309ec0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGPIPE: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGILL: [libjvm.so+0x309ec0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: [libjvm.so+0x30bef0], sa_mask[0]=0x00000000, sa_flags=0x10000004
SIGHUP: [libjvm.so+0x30b910], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGINT: [libjvm.so+0x30b910], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x30b910], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGTERM: [libjvm.so+0x30b910], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGUSR2: [libjvm.so+0x30bef0], sa_mask[0]=0x00000000, sa_flags=0x10000004


---------------  S Y S T E M  ---------------

OS:squeeze/sid

uname:Linux 2.6.35-23-generic #41-Ubuntu SMP Wed Nov 24 11:55:36 UTC 2010 x86_64
libc:glibc 2.12.1 NPTL 2.12.1 
rlimit: STACK 8192k, CORE 0k, NPROC infinity, NOFILE 1024, AS infinity
load average:0.67 0.54 0.36

CPU:total 8 (8 cores per cpu, 2 threads per core) family 6 model 10 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, ht

Memory: 4k page, physical 8191552k(3359308k free), swap 1016828k(1016828k free)

vm_info: Java HotSpot(TM) Client VM (1.6.0_03-b05) for linux-x86, built on Sep 24 2007 22:45:46 by "java_re" with gcc 3.2.1-7a (J2SE release)
Community
  • 1
  • 1
Paul J. Lucas
  • 6,895
  • 6
  • 44
  • 88
  • I cannot comment on how to debug, but contrarily to popular belief, JVM crashes are exceedingly common. Here's a reproducible one on a rock-stable solid system that regularly reaches uptime of 6 months (it's my workstation, so I don't mind rebooting it once ever six months, when there's a brow-out ;) http://stackoverflow.com/questions/2299250 – SyntaxT3rr0r Dec 23 '10 at 17:54
  • 2
    That's a great article you linked there: http://weblogs.java.net/blog/kohsuke/archive/2009/02/crash_course_on.html Instead of taking a *"JVM do not crash, you're hardware is faulty"* kind of knee-jerk approach, the guy does explain very nicely how to get down to the machine code and **proves** that the JVM is indeed faulty. +1 to your question and don't pay attention to the ones who're going to say here that *"JVMs do not crash anymore"*. – SyntaxT3rr0r Dec 23 '10 at 17:58
  • 1
    Unless there is sensitive information in the crash, then add the crash dump text to your question. It tells a lot about what happened. – Thorbjørn Ravn Andersen Dec 23 '10 at 18:27
  • 1
    @SpoonBender. In my experience JVM crashes are closely related to JNI. – Thorbjørn Ravn Andersen Dec 23 '10 at 18:29
  • As @Thorbjørn Ravn Andersen mentioned, start with the data dump. Start by googling it to see if it is a reported but (I used to be able to get Java 1.4.2 to crash with a known bug pretty regularly that I had to work around). If you come up empty, bring the dump here. – justkt Dec 23 '10 at 19:11

3 Answers3

4

Unless your JNI library has corrupted a random area of memory (which is extreamly difficult to debug), the most likely cause of a crash in the libjvm is a bug in the JVM. Given the current release is Java 6 update 23 and you have Java 6 update 3, upgrading is the first thing I would try.

BTW: Also you appear to have a 64-bit OS, you might like to try the 64-bit Java.

BTW2: You can write -Xmx2000000k as -mx2g. The default maximum on a 8 GB machine should be about 2 GB anyway.

Peter Lawrey
  • 525,659
  • 79
  • 751
  • 1,130
  • 2
    +1 for trying a new JVM first, especially if this crash is repeatable and easy to reproduce. It may be faster to upgrade and see if it happens than track down the cause in an older JVM. – AngerClown Dec 24 '10 at 01:48
2

I was expecting a JNI call in the crash dump, but it does not appear to be the cause, but instead a bug in the pthreads library.

I cannot remember offhand if the pthreads library must be provided by the underlying platform or it is baked into the JRE, but I would suggest that you ensure that you are running on one of the officially supported Linux platforms and if not, then strongly consider switching.

Thorbjørn Ravn Andersen
  • 73,784
  • 33
  • 194
  • 347
1

Do you really need -Xmx2000000k? Looks like you are only using about 84MB of that.

My experience is that you really can't reliably go above about 1700MB heap in a 32 bit process space anyway. So, that setting may be causing this problem.

AngerClown
  • 6,149
  • 1
  • 25
  • 28