3

The application is basically a Servlet on top of Apaches Tomcat 8 and crashes every 3 to n days. The most interesting part is, that the last 3 crashes happened at 1.30 PM + 3 to 4 seconds. Every other crash happens at 15, 30, 45 or the full hour. The same application worked for months on Windows 7 with Java 7 something.

I found this report so I checked the available space at /tmp and I have plenty of space here. However, the next time I ran the VM with:

-XX:-UsePerfData

-XX:+PerfDisableSharedMem

and the hsperfdata file is no longer written but the problem persist. According to this report PerfData might be still used to some extend even if it is disabled.

It seems to be this piece of code:

  void PerfLongVariant::sample() {

  // JJJ - This should not happen.  Maybe the first sample is taken
  // while the _sample_helper is being null'ed out.
  // assert(_sample_helper != NULL || _sampled != NULL, "unexpected state");
  if (_sample_helper == NULL) return;

  if (_sample_helper != NULL) {
    *(jlong*)_valuep = _sample_helper->take_sample();
  }
  else if (_sampled != NULL) {
    *(jlong*)_valuep = *_sampled;
  }
}

I assume the problem occures here:

*(jlong*)_valuep = _sample_helper->take_sample();

but I am not sure which implementation is used at runtime:

statSampler seems unlikely as I don't see the connection to elapsed_counter except for the above mentioned phenomena with the crash time.

/*
 * helper class to provide for sampling of the elapsed_counter value
 * maintained in the OS class.
 */
class HighResTimeSampler : public PerfSampleHelper {
  public:
    jlong take_sample() { return os::elapsed_counter(); }
};

or gSpaceCounter

 inline jlong take_sample() {
  return _gen->used();
}

and spaceCounters and cSpaceCounter I have the impression that the size of one of the pools in the Java memory pool is returned before it is updated. Which ultimately lead to SIGBUS 7, bad memory access.

What are my options?

The last crash report (dynamic libraries removed):

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f463c87a82e, pid=10876, tid=0x00007f4624cfc700
#
# JRE version: OpenJDK Runtime Environment (8.0_101-b13) (build 1.8.0_101-b13)
# Java VM: OpenJDK 64-Bit Server VM (25.101-b13 mixed mode linux-amd64 compressed oops)
# Derivative: IcedTea 3.1.0
# Distribution: Custom build (Fri Jul 29 23:44:32 UTC 2016)
# Problematic frame:
# V  [libjvm.so+0x7f582e]  PerfLongVariant::sample()+0x1e
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   http://icedtea.classpath.org/bugzilla
#

---------------  T H R E A D  ---------------

Current thread (0x00007f46340ce800):  WatcherThread [stack: 0x00007f4624bfc000,0x00007f4624cfd000] [id=10909]

siginfo: si_signo: 7 (SIGBUS), si_code: 2 (BUS_ADRERR), si_addr: 0x00007f463dc89420

Registers:
RAX=0x0000000007114e40, RBX=0x00007f463dc89420, RCX=0x0000000000000000, RDX=0x00007f463ce2d210
RSP=0x00007f4624cfbe00, RBP=0x00007f4624cfbe10, RSI=0x0000000000000032, RDI=0x00007f4634018fa0
R8 =0x00007f463ce69a48, R9 =0x0000000000000001, R10=0x00182e09a9af66e8, R11=0x0000000000000000
R12=0x0000000000000008, R13=0x00007f463401b1a0, R14=0x0000000000000002, R15=0x00007f463ce6f3c0
RIP=0x00007f463c87a82e, EFLAGS=0x0000000000010202, CSGSFS=0x0000000000000033, ERR=0x0000000000000007
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f4624cfbe00)
0x00007f4624cfbe00:   00007f46340ce800 0000000000000001
0x00007f4624cfbe10:   00007f4624cfbe40 00007f463c8ecd44
0x00007f4624cfbe20:   00007f4624cfbe40 0000000000000001
0x00007f4624cfbe30:   0000000000000032 0000000000000032
0x00007f4624cfbe40:   00007f4624cfbe90 00007f463c92e380
0x00007f4624cfbe50:   00007f46340cfad0 00007f4634007630
0x00007f4624cfbe60:   0000000000000000 0000000000000032
0x00007f4624cfbe70:   00007f46340ce800 00007f46340ce800
0x00007f4624cfbe80:   0000000000000000 00007f4624cfc700
0x00007f4624cfbe90:   00007f4624cfbf10 00007f463c954bf7
0x00007f4624cfbea0:   0000000000000000 0000000000000000
0x00007f4624cfbeb0:   0000000000000001 00007f46340ce800
0x00007f4624cfbec0:   00007f4624cfbf10 00007f463c80b8cf
0x00007f4624cfbed0:   0000000000000000 00007f46340cfbc0
0x00007f4624cfbee0:   00007f46340cfad0 00007f46340cfbc0
0x00007f4624cfbef0:   00007f46340cfad0 00007f46340ce800
0x00007f4624cfbf00:   0000000000000000 00007f4624cfc700
0x00007f4624cfbf10:   00007f4624cfbf50 00007f463c83e6d2
0x00007f4624cfbf20:   0000000000000000 0000000000000000
0x00007f4624cfbf30:   0000000000000000 00007f463dc70b5f
0x00007f4624cfbf40:   0000000000000000 00007f4624cfc700
0x00007f4624cfbf50:   0000000000000000 00007f463ce92734
0x00007f4624cfbf60:   0000000000000000 00007f4624cfc700
0x00007f4624cfbf70:   00007f4624cfc700 c5fe0a2abaafece4
0x00007f4624cfbf80:   0000000000000000 00007f463dc70b5f
0x00007f4624cfbf90:   0000000000000000 00007f4624cfc700
0x00007f4624cfbfa0:   3b7243b5c46fece4 3b7273f8f755ece4
0x00007f4624cfbfb0:   0000000000000000 0000000000000000
0x00007f4624cfbfc0:   0000000000000000 0000000000000000
0x00007f4624cfbfd0:   0000000000000000 0000000000000000
0x00007f4624cfbfe0:   0000000000000000 0000000000000000
0x00007f4624cfbff0:   00007f4624cfc700 00007f463d5aad3d 

Instructions: (pc=0x00007f463c87a82e)
0x00007f463c87a80e:   a0 ff 48 8b 47 38 48 85 c0 74 1e 55 48 89 e5 53
0x00007f463c87a81e:   48 83 ec 08 48 8b 5f 28 48 8b 10 48 89 c7 ff 12
0x00007f463c87a82e:   48 89 03 48 83 c4 08 5b 5d f3 c3 90 66 0f 1f 44
0x00007f463c87a83e:   00 00 55 48 8d 05 08 02 5b 00 48 89 e5 41 57 41 




Stack: [0x00007f4624bfc000,0x00007f4624cfd000],  sp=0x00007f4624cfbe00,  free space=1023k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x7f582e]  PerfLongVariant::sample()+0x1e
V  [libjvm.so+0x867d44]  StatSampler::sample_data(PerfDataList*)+0x34
V  [libjvm.so+0x8a9380]  PeriodicTask::real_time_tick(int)+0x90
V  [libjvm.so+0x8cfbf7]  WatcherThread::run()+0x47
V  [libjvm.so+0x7b96d2]  java_start(Thread*)+0xe2


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00007f45f0011000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=15853, stack(0x00007f45e72f9000,0x00007f45e73fa000)]
  0x00007f45f0010000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=15852, stack(0x00007f45f48ce000,0x00007f45f49cf000)]
  0x00007f45f0018800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=15851, stack(0x00007f45e75fa000,0x00007f45e76fb000)]
  0x00007f4600699000 JavaThread "BackendCache" daemon [_thread_blocked, id=15363, stack(0x00007f45e76fb000,0x00007f45e77fc000)]
  0x00007f46352da800 JavaThread "Thread-9" daemon [_thread_blocked, id=15362, stack(0x00007f45e77fc000,0x00007f45e78fd000)]
  0x00007f45ea07c800 JavaThread "Timer-0" daemon [_thread_blocked, id=15356, stack(0x00007f45f4063000,0x00007f45f4164000)]
  0x00007f4634dd2000 JavaThread "Thread-4" daemon [_thread_blocked, id=15355, stack(0x00007f45f4564000,0x00007f45f4665000)]
  0x00007f45ec04b000 JavaThread "http-apr-8080-exec-10" daemon [_thread_blocked, id=11253, stack(0x00007f45f49cf000,0x00007f45f4ad0000)]
  0x00007f45f000b000 JavaThread "http-apr-8080-exec-9" daemon [_thread_blocked, id=11252, stack(0x00007f45f4ad0000,0x00007f45f4bd1000)]
  0x00007f45f0009800 JavaThread "http-apr-8080-exec-8" daemon [_thread_blocked, id=11251, stack(0x00007f45f4bd1000,0x00007f45f4cd2000)]
  0x00007f45f0007800 JavaThread "http-apr-8080-exec-7" daemon [_thread_blocked, id=11250, stack(0x00007f45f4cd2000,0x00007f45f4dd3000)]
  0x00007f45f0006000 JavaThread "http-apr-8080-exec-6" daemon [_thread_blocked, id=11249, stack(0x00007f45f4dd3000,0x00007f45f4ed4000)]
  0x00007f45ec03f800 JavaThread "http-apr-8080-exec-5" daemon [_thread_blocked, id=11248, stack(0x00007f45f4ed4000,0x00007f45f4fd5000)]
  0x00007f45f0004000 JavaThread "http-apr-8080-exec-4" daemon [_thread_blocked, id=11247, stack(0x00007f45f4fd5000,0x00007f45f50d6000)]
  0x00007f45f0002800 JavaThread "http-apr-8080-exec-3" daemon [_thread_blocked, id=10969, stack(0x00007f45f50d6000,0x00007f45f51d7000)]
  0x00007f45ec079000 JavaThread "http-apr-8080-exec-2" daemon [_thread_blocked, id=10968, stack(0x00007f45f60c4000,0x00007f45f61c5000)]
  0x00007f45f0001800 JavaThread "http-apr-8080-exec-1" daemon [_thread_blocked, id=10967, stack(0x00007f45f5fc3000,0x00007f45f60c4000)]
  0x00007f463459b800 JavaThread "ajp-apr-8009-AsyncTimeout" daemon [_thread_blocked, id=10930, stack(0x00007f45f53d7000,0x00007f45f54d8000)]
  0x00007f4634599800 JavaThread "ajp-apr-8009-Acceptor-0" daemon [_thread_in_native, id=10929, stack(0x00007f45f54d8000,0x00007f45f55d9000)]
  0x00007f4634597800 JavaThread "ajp-apr-8009-Poller" daemon [_thread_blocked, id=10928, stack(0x00007f45f55d9000,0x00007f45f56da000)]
  0x00007f4634595800 JavaThread "http-apr-8080-AsyncTimeout" daemon [_thread_blocked, id=10927, stack(0x00007f45f56da000,0x00007f45f57db000)]
  0x00007f4634594800 JavaThread "http-apr-8080-Acceptor-0" daemon [_thread_in_native, id=10926, stack(0x00007f45f59db000,0x00007f45f5adc000)]
  0x00007f4634592800 JavaThread "http-apr-8080-Sendfile" daemon [_thread_blocked, id=10925, stack(0x00007f45f5adc000,0x00007f45f5bdd000)]
  0x00007f4634591000 JavaThread "http-apr-8080-Poller" daemon [_thread_blocked, id=10924, stack(0x00007f45f5c94000,0x00007f45f5d95000)]
  0x00007f463458d800 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=10923, stack(0x00007f45f5ec2000,0x00007f45f5fc3000)]
  0x00007f4634515800 JavaThread "GC Daemon" daemon [_thread_blocked, id=10911, stack(0x00007f45f7434000,0x00007f45f7535000)]
  0x00007f4634145800 JavaThread "AsyncFileHandlerWriter-1878246837" daemon [_thread_blocked, id=10910, stack(0x00007f4624afb000,0x00007f4624bfc000)]
  0x00007f46340cc000 JavaThread "Service Thread" daemon [_thread_blocked, id=10908, stack(0x00007f4624cfd000,0x00007f4624dfe000)]
  0x00007f46340c7800 JavaThread "C1 CompilerThread1" daemon [_thread_blocked, id=10907, stack(0x00007f4624dfe000,0x00007f4624eff000)]
  0x00007f46340ba000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=10906, stack(0x00007f4624eff000,0x00007f4625000000)]
  0x00007f46340b8000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=10905, stack(0x00007f4638052000,0x00007f4638153000)]
  0x00007f463408d800 JavaThread "Finalizer" daemon [_thread_blocked, id=10904, stack(0x00007f4638197000,0x00007f4638298000)]
  0x00007f4634089000 JavaThread "Reference Handler" daemon [_thread_blocked, id=10903, stack(0x00007f4638298000,0x00007f4638399000)]
  0x00007f4634008000 JavaThread "main" [_thread_in_native, id=10899, stack(0x00007f463db71000,0x00007f463dc72000)]

Other Threads:
  0x00007f463407f800 VMThread [stack: 0x00007f4638399000,0x00007f463849a000] [id=10902]
=>0x00007f46340ce800 WatcherThread [stack: 0x00007f4624bfc000,0x00007f4624cfd000] [id=10909]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00007f4634007630] PeriodicTask_lock - owner thread: 0x00007f46340ce800

Heap:
 PSYoungGen      total 161792K, used 115891K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
  eden space 161280K, 71% used [0x00000000f6180000,0x00000000fd294e40,0x00000000fff00000)
  from space 512K, 18% used [0x00000000fff00000,0x00000000fff18020,0x00000000fff80000)
  to   space 512K, 0% used [0x00000000fff80000,0x00000000fff80000,0x0000000100000000)
 ParOldGen       total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
  object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
 Metaspace       used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
  class space    used 3342K, capacity 3473K, committed 3584K, reserved 1048576K

Card table byte_map: [0x00007f463aaea000,0x00007f463abd9000] byte_map_base: 0x00007f463a3d8000

Marking Bits: (ParMarkBitMap*) 0x00007f463ce6d740
 Begin Bits: [0x00007f4639a08000, 0x00007f463a178000)
 End Bits:   [0x00007f463a178000, 0x00007f463a8e8000)

Polling page: 0x00007f463dc91000

CodeCache: size=245760Kb used=13583Kb max_used=14910Kb free=232176Kb
 bounds [0x00007f4625000000, 0x00007f4625ec0000, 0x00007f4634000000]
 total_blobs=4397 nmethods=3905 adapters=407
 compilation: enabled

GC Heap History (10 events):
Event: 241117.477 GC heap before
{Heap before GC invocations=451 (full 2):
 PSYoungGen      total 161792K, used 161456K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
  eden space 161280K, 100% used [0x00000000f6180000,0x00000000fff00000,0x00000000fff00000)
  from space 512K, 34% used [0x00000000fff80000,0x00000000fffac030,0x0000000100000000)
  to   space 512K, 0% used [0x00000000fff00000,0x00000000fff00000,0x00000000fff80000)
 ParOldGen       total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
  object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
 Metaspace       used 31406K, capacity 31972K, committed 32256K, reserved 1077248K
  class space    used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
Event: 241117.480 GC heap after
Heap after GC invocations=451 (full 2):
 PSYoungGen      total 161792K, used 128K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
  eden space 161280K, 0% used [0x00000000f6180000,0x00000000f6180000,0x00000000fff00000)
  from space 512K, 25% used [0x00000000fff00000,0x00000000fff20020,0x00000000fff80000)
  to   space 512K, 0% used [0x00000000fff80000,0x00000000fff80000,0x0000000100000000)
 ParOldGen       total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
  object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
 Metaspace       used 31406K, capacity 31972K, committed 32256K, reserved 1077248K
  class space    used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
}

Event: 241985.401 GC heap before
{Heap before GC invocations=453 (full 2):
 PSYoungGen      total 161792K, used 161408K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
  eden space 161280K, 100% used [0x00000000f6180000,0x00000000fff00000,0x00000000fff00000)
  from space 512K, 25% used [0x00000000fff80000,0x00000000fffa0020,0x0000000100000000)
  to   space 512K, 0% used [0x00000000fff00000,0x00000000fff00000,0x00000000fff80000)
 ParOldGen       total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
  object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
 Metaspace       used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
  class space    used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
Event: 241985.404 GC heap after
Heap after GC invocations=453 (full 2):
 PSYoungGen      total 161792K, used 96K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
  eden space 161280K, 0% used [0x00000000f6180000,0x00000000f6180000,0x00000000fff00000)
  from space 512K, 18% used [0x00000000fff00000,0x00000000fff18020,0x00000000fff80000)
  to   space 512K, 0% used [0x00000000fff80000,0x00000000fff80000,0x0000000100000000)
 ParOldGen       total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
  object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
 Metaspace       used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
  class space    used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
}
Event: 242421.299 GC heap before
{Heap before GC invocations=454 (full 2):
 PSYoungGen      total 161792K, used 161376K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
  eden space 161280K, 100% used [0x00000000f6180000,0x00000000fff00000,0x00000000fff00000)
  from space 512K, 18% used [0x00000000fff00000,0x00000000fff18020,0x00000000fff80000)
  to   space 512K, 0% used [0x00000000fff80000,0x00000000fff80000,0x0000000100000000)
 ParOldGen       total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
  object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
 Metaspace       used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
  class space    used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
Event: 242421.301 GC heap after
Heap after GC invocations=454 (full 2):
 PSYoungGen      total 161792K, used 112K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
  eden space 161280K, 0% used [0x00000000f6180000,0x00000000f6180000,0x00000000fff00000)
  from space 512K, 21% used [0x00000000fff80000,0x00000000fff9c010,0x0000000100000000)
  to   space 512K, 0% used [0x00000000fff00000,0x00000000fff00000,0x00000000fff80000)
 ParOldGen       total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
  object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
 Metaspace       used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
  class space    used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
}
Event: 242856.234 GC heap before
{Heap before GC invocations=455 (full 2):
 PSYoungGen      total 161792K, used 161392K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
  eden space 161280K, 100% used [0x00000000f6180000,0x00000000fff00000,0x00000000fff00000)
  from space 512K, 21% used [0x00000000fff80000,0x00000000fff9c010,0x0000000100000000)
  to   space 512K, 0% used [0x00000000fff00000,0x00000000fff00000,0x00000000fff80000)
 ParOldGen       total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
  object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
 Metaspace       used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
  class space    used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
Event: 242856.236 GC heap after
Heap after GC invocations=455 (full 2):
 PSYoungGen      total 161792K, used 96K [0x00000000f6180000, 0x0000000100000000, 0x0000000100000000)
  eden space 161280K, 0% used [0x00000000f6180000,0x00000000f6180000,0x00000000fff00000)
  from space 512K, 18% used [0x00000000fff00000,0x00000000fff18020,0x00000000fff80000)
  to   space 512K, 0% used [0x00000000fff80000,0x00000000fff80000,0x0000000100000000)
 ParOldGen       total 36352K, used 23540K [0x00000000e2400000, 0x00000000e4780000, 0x00000000f6180000)
  object space 36352K, 64% used [0x00000000e2400000,0x00000000e3afd378,0x00000000e4780000)
 Metaspace       used 31407K, capacity 31972K, committed 32256K, reserved 1077248K
  class space    used 3342K, capacity 3473K, committed 3584K, reserved 1048576K
}


Internal exceptions (10 events):
Event: 57534.673 Thread 0x00007f4634dd2000 Exception <a 'java/lang/ClassNotFoundException': org/hibernate/util/CloneableCustomizer> (0x00000000faf05e08) thrown at [/home/abuild/rpmbuild/BUILD/icedtea-3.1.0/openjdk/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 57534.674 Thread 0x00007f4634dd2000 Exception <a 'java/lang/ClassNotFoundException': org/hibernate/event/EventListenersCustomizer> (0x00000000faf3abc8) thrown at [/home/abuild/rpmbuild/BUILD/icedtea-3.1.0/openjdk/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 210]
Event: 57540.284 Thread 0x00007f4634bee000 Exception <a 'java/net/SocketTimeoutException': connect timed out> (0x00000000fcb2fac8) thrown at [/home/abuild/rpmbuild/BUILD/icedtea-3.1.0/openjdk/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 57943.337 Thread 0x00007f46352da800 Implicit null exception at 0x00007f4625d42a9f to 0x00007f4625d4931d
Event: 57943.340 Thread 0x00007f4600699000 Implicit null exception at 0x00007f4625d42a9f to 0x00007f4625d4931d
Event: 58820.887 Thread 0x00007f45ea078800 Implicit null exception at 0x00007f462514eab4 to 0x00007f462514f049
Event: 61135.304 Thread 0x00007f4634dd2000 Implicit null exception at 0x00007f46258c9192 to 0x00007f46258ca9a4
Event: 61135.314 Thread 0x00007f4634dd2000 Implicit null exception at 0x00007f46256f414e to 0x00007f46256f4371
Event: 61135.315 Thread 0x00007f4634dd2000 Implicit null exception at 0x00007f4625902079 to 0x00007f4625903285
Event: 63765.574 Thread 0x00007f45f0012800 Exception <a 'java/lang/InterruptedException'> (0x00000000fb423328) thrown at [/home/abuild/rpmbuild/BUILD/icedtea-3.1.0/openjdk/hotspot/src/share/vm/runtime/objectMonitor.cpp, line 1495]


VM Arguments:
jvm_args: -Djavax.sql.DataSource.Factory=org.apache.commons.dbcp.BasicDataSourceFactory -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 
java_command: org.apache.catalina.startup.Bootstrap start
java_class_path (initial): /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/lib/jvm/jre
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
SHELL=/bin/sh

---------------  S Y S T E M  ---------------

OS:SUSE Linux Enterprise Server 12 (x86_64)
VERSION = 12
PATCHLEVEL = 2
 
uname:Linux 4.4.49-92.11-default #1 SMP Fri Feb 17 08:29:30 UTC 2017 (8f9478a) x86_64
libc:glibc 2.22 NPTL 2.22 
rlimit: STACK 8192k, CORE 0k, NPROC 7522, NOFILE 4096, AS infinity
load average:0.00 0.00 0.00

/proc/meminfo:
MemTotal:        1943796 kB
MemFree:          117384 kB
MemAvailable:    1056832 kB
Buffers:               8 kB
Cached:          1116816 kB
SwapCached:         3152 kB
Active:          1430656 kB
Inactive:         284260 kB
Active(anon):     438228 kB
Inactive(anon):   233656 kB
Active(file):     992428 kB
Inactive(file):    50604 kB
Unevictable:          80 kB
Mlocked:              80 kB
SwapTotal:       4187132 kB
SwapFree:        4181900 kB
Dirty:               156 kB
Writeback:             0 kB
AnonPages:        595020 kB
Mapped:            68856 kB
Shmem:             73792 kB
Slab:              75320 kB
SReclaimable:      55520 kB
SUnreclaim:        19800 kB
KernelStack:        3664 kB
PageTables:         5648 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     5159028 kB
Committed_AS:    1247032 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:    471040 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:      366528 kB
DirectMap2M:     1730560 kB


CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 37 stepping 1, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, aes, clmul, tsc, tscinvbit

/proc/cpuinfo:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model  : 37
model name : Intel(R) Xeon(R) CPU           E5645  @ 2.40GHz
stepping : 1
microcode : 0x13
cpu MHz  : 2400.085
cache size : 12288 KB
physical id : 0
siblings : 1
core id  : 0
cpu cores : 1
apicid  : 0
initial apicid : 0
fpu  : yes
fpu_exception : yes
cpuid level : 11
wp  : yes
flags  : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm ida arat epb dtherm
bugs  :
bogomips : 4800.17
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model  : 37
model name : Intel(R) Xeon(R) CPU           E5645  @ 2.40GHz
stepping : 1
microcode : 0x13
cpu MHz  : 2400.085
cache size : 12288 KB
physical id : 2
siblings : 1
core id  : 0
cpu cores : 1
apicid  : 2
initial apicid : 2
fpu  : yes
fpu_exception : yes
cpuid level : 11
wp  : yes
flags  : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts nopl xtopology tsc_reliable nonstop_tsc aperfmperf pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm ida arat epb dtherm
bugs  :
bogomips : 4800.17
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 1943796k(117384k free), swap 4187132k(4181900k free)

vm_info: OpenJDK 64-Bit Server VM (25.101-b13) for linux-amd64 JRE (1.8.0_101-b13), built on Jul 30 2016 00:00:27 by "abuild" with gcc 4.8.5

time: Sat Nov 11 13:30:04 2017
elapsed time: 243167 seconds (2d 19h 32m 47s)

Register to memory mapping:

RAX=0x0000000007114e40 is an unknown value
RBX=0x00007f463dc89420 is an unknown value
RCX=0x0000000000000000 is an unknown value
RDX=0x00007f463ce2d210: <offset 0xda8210> in /usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/amd64/server/libjvm.so at 0x00007f463c085000
RSP=0x00007f4624cfbe00 is an unknown value
RBP=0x00007f4624cfbe10 is an unknown value
RSI=0x0000000000000032 is an unknown value
RDI=0x00007f4634018fa0 is an unknown value
R8 =0x00007f463ce69a48: <offset 0xde4a48> in /usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/amd64/server/libjvm.so at 0x00007f463c085000
R9 =0x0000000000000001 is an unknown value
R10=0x00182e09a9af66e8 is an unknown value
R11=0x0000000000000000 is an unknown value
R12=0x0000000000000008 is an unknown value
R13=0x00007f463401b1a0 is an unknown value
R14=0x0000000000000002 is an unknown value
R15=0x00007f463ce6f3c0: <offset 0xdea3c0> in /usr/lib64/jvm/java-1.8.0-openjdk-1.8.0/jre/lib/amd64/server/libjvm.so at 0x00007f463c085000
Community
  • 1
  • 1
steffan
  • 619
  • 1
  • 9
  • 18
  • 1
    to be honest there is just one user here that I am aware of that can actually might have some valuable input for you; as such editing to include the `jvm` tag as I know he watches it. – Eugene Nov 13 '17 at 21:50
  • 1) Memory map is the essential part of error report, you'd better not remove it. SIGBUS (rather than SIGSEGV) indicates that the target address is mapped, so it is good to know where. 2) There is no `PerfDisableSharedMem` among the arguments. 3) A typical case for SIGBUS is when a mapped file is truncated or when the underlying storage becomes unaccessible. 4) Check if there are periodic tasks in the system to clean /tmp or something of this kind. – apangin Nov 13 '17 at 22:39
  • 1) Added 2) True, I I added the parameters in the setenv.sh, but that was the wrong location... In my environment I can see them as VM Arguments. 3) Okay. 4) We checked for cronjobs etc. and found nothing suspicious. – steffan Nov 20 '17 at 20:45
  • We are running into the same problem with SLES12SP2 kernel version 4.4.90+. It may have something to do with file system(btrfs) falsely reports out of disk space and triggers all running 1.8.0_151 JVM (openjdk and oracle) to crash with sigbus. Going back to kernel 4.4.59 fixes the issue – Dan Tran Dec 30 '17 at 03:54
  • 1
    Starting the JVM with -XX:-UsePerfData and -XX:+PerfDisableSharedMem solved the problem so far (no problem for over a month). Could be a SLES12SP2 specific problem indeed, because the problem occurs with different JVM versions. – steffan Dec 30 '17 at 20:23
  • Steffan's suggested workaround seems to work, however, we are now facing rabbitmq crashing – Dan Tran Jan 08 '18 at 23:30
  • sles12sp2 kernel 4.4.114-92.64.1 has the fix – Dan Tran Feb 20 '18 at 03:06

0 Answers0