2

I am currently working on a program that I run within Jetty. Sometimes (like once in 2 weeks) it crashes the JVM and I'm not sure why this is happening. My only guess is that it could be memory-related. Since the error pesists already for a while I had updated the JRE from 8.0_66 to 8.0_77 a while ago.

It generates a hs_err_pid logfile:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000000038ce802, pid=2808, tid=3456
#
# JRE version: Java(TM) SE Runtime Environment (8.0_77-b03) (build 1.8.0_77-b03)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.77-b03 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# J 5131 C2 java.util.HashSet.add(Ljava/lang/Object;)Z (20 bytes) @ 0x00000000038ce802 [0x00000000038ce720+0xe2]
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00000000699ee800):  JavaThread "Atmosphere-Scheduler-5956" daemon [_thread_in_Java, id=3456, stack(0x0000000058280000,0x0000000058380000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x00000000ff7f000c

Registers:
RAX=0x0000000064388461, RBX=0x00000000ff7f0000, RCX=0x00000000c03d0988, RDX=0x00000000f0847b40
RSP=0x000000005837f700, RBP=0x00000000c03d09bc, RSI=0x00000000f0847b70, RDI=0x00000000147fc2c8
R8 =0x0000000000000010, R9 =0x000000006438e059, R10=0x0000000000000009, R11=0x00000000c03d0948
R12=0x0000000000000000, R13=0x00000000f0847b40, R14=0x00000000f0847b40, R15=0x00000000699ee800
RIP=0x00000000038ce802, EFLAGS=0x0000000000010286

Top of Stack: (sp=0x000000005837f700)
0x000000005837f700:   00000000f0847b40 00000000c03d0948
0x000000005837f710:   00000000f0847b70 00000000f0847d30
0x000000005837f720:   0000000000000002 000007feff573356
0x000000005837f730:   0000002200000019 00000000c03d0470
0x000000005837f740:   00000000c006b7b0 000000007483417f
0x000000005837f750:   00000000f0847b70 0000000003ad3ed4
0x000000005837f760:   00000000c03d0470 00000000ff3525a0
0x000000005837f770:   00000000c03d0400 0000000003b6d636
0x000000005837f780:   00000000c03d03a0 00000000f0847b40
0x000000005837f790:   00000000147e6e68 00000000c03d0908
0x000000005837f7a0:   00000000f0847b70 00000000c03d0908
0x000000005837f7b0:   00000000c03d0618 00000000c03d0608
0x000000005837f7c0:   00000000c03d0520 000000007483421c
0x000000005837f7d0:   0000004195065501 00000000ffffffff
0x000000005837f7e0:   0000000000000000 00000000712a65c6
0x000000005837f7f0:   00000000c03d0470 0000000003b64ae8 

Instructions: (pc=0x00000000038ce802)
0x00000000038ce7e2:   23 d1 45 3b d0 0f 83 f0 0a 00 00 4a 8d 6c 91 10
0x00000000038ce7f2:   8b 5d 00 bf c8 c2 7f 14 85 db 0f 84 0b 01 00 00
0x00000000038ce802:   8b 53 0c 48 8b f3 41 3b d1 0f 84 0b 05 00 00 44
0x00000000038ce812:   8b 53 08 41 81 fa f0 65 81 14 0f 85 83 05 00 00 


Register to memory mapping:

RAX=0x0000000064388461 is an unknown value
RBX=0x00000000ff7f0000 is an unallocated location in the heap
RCX=0x00000000c03d0988 is an oop
[Ljava.util.HashMap$Node; 
 - klass: 'java/util/HashMap$Node'[]
 - length: 16
RDX=0x00000000f0847b40 is an oop
java.util.concurrent.ThreadPoolExecutor$Worker 
 - klass: 'java/util/concurrent/ThreadPoolExecutor$Worker'
RSP=0x000000005837f700 is pointing into the stack for thread: 0x00000000699ee800
RBP=0x00000000c03d09bc is pointing into object: 0x00000000c03d0988
[Ljava.util.HashMap$Node; 
 - klass: 'java/util/HashMap$Node'[]
 - length: 16
RSI=0x00000000f0847b70 is an oop
java.lang.Thread 
 - klass: 'java/lang/Thread'
RDI=0x00000000147fc2c8 is pointing into metadata
R8 =0x0000000000000010 is an unknown value
R9 =0x000000006438e059 is an unknown value
R10=0x0000000000000009 is an unknown value
R11=0x00000000c03d0948 is an oop
java.util.HashMap 
 - klass: 'java/util/HashMap'
R12=0x0000000000000000 is an unknown value
R13=0x00000000f0847b40 is an oop
java.util.concurrent.ThreadPoolExecutor$Worker 
 - klass: 'java/util/concurrent/ThreadPoolExecutor$Worker'
R14=0x00000000f0847b40 is an oop
java.util.concurrent.ThreadPoolExecutor$Worker 
 - klass: 'java/util/concurrent/ThreadPoolExecutor$Worker'
R15=0x00000000699ee800 is a thread


Stack: [0x0000000058280000,0x0000000058380000],  sp=0x000000005837f700,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0x00000000038ce802


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x00000000699ed000 JavaThread "Atmosphere-Scheduler-5938" daemon [_thread_blocked, id=4788, stack(0x0000000057470000,0x0000000057570000)]
=>0x00000000699ee800 JavaThread "Atmosphere-Scheduler-5956" daemon [_thread_in_Java, id=3456, stack(0x0000000058280000,0x0000000058380000)]
  0x00000000699eb800 JavaThread "Atmosphere-Scheduler-5936" daemon [_thread_blocked, id=3520, stack(0x000000005afa0000,0x000000005b0a0000)]
  0x0000000066296800 JavaThread "Atmosphere-Scheduler-5953" daemon [_thread_blocked, id=2076, stack(0x000000006a360000,0x000000006a460000)]
  0x0000000056e8e800 JavaThread "qtp1642360923-8426" [_thread_blocked, id=4740, stack(0x0000000058070000,0x0000000058170000)]
  0x0000000056e8b800 JavaThread "qtp1642360923-8425" [_thread_blocked, id=4684, stack(0x000000006ff20000,0x0000000070020000)]
  0x0000000056e92000 JavaThread "qtp1642360923-8424" [_thread_blocked, id=4676, stack(0x000000006f820000,0x000000006f920000)]
  0x000000006c387000 JavaThread "qtp1642360923-8121" [_thread_blocked, id=4344, stack(0x0000000070250000,0x0000000070350000)]
  0x000000006c385800 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=3212, stack(0x00000000700f0000,0x00000000701f0000)]
  0x000000005a1b5000 JavaThread "MySQL Statement Cancellation Timer" daemon [_thread_blocked, id=4636, stack(0x0000000059ea0000,0x0000000059fa0000)]
  0x000000005a1b6800 JavaThread "Thread-14" [_thread_blocked, id=4620, stack(0x0000000058990000,0x0000000058a90000)]
  0x000000005a1b9800 JavaThread "DestroyJavaVM" [_thread_blocked, id=3172, stack(0x00000000025c0000,0x00000000026c0000)]
  0x000000005a1b8800 JavaThread "Scanner-0" daemon [_thread_blocked, id=4816, stack(0x000000006e0b0000,0x000000006e1b0000)]
  0x000000005a1b3800 JavaThread "Scheduler-358699161" [_thread_blocked, id=4732, stack(0x000000006a050000,0x000000006a150000)]
  0x0000000057928000 JavaThread "qtp1642360923-21-selector-ServerConnectorManager@11b27ced/0" [_thread_in_native, id=4248, stack(0x0000000058a90000,0x0000000058b90000)]
  0x0000000057927800 JavaThread "qtp1642360923-20-acceptor-0@3626666f-ServerConnector@48974e45{SSL-http/1.1}{0.0.0.0:8443}" [_thread_in_native, id=4244, stack(0x0000000058880000,0x0000000058980000)]
  0x0000000057924800 JavaThread "qtp1642360923-19-selector-ServerConnectorManager@57c6d586/0" [_thread_in_native, id=4240, stack(0x00000000570b0000,0x00000000571b0000)]
  0x0000000057922000 JavaThread "qtp1642360923-18-selector-ServerConnectorManager@11b27ced/1" [_thread_in_native, id=4236, stack(0x00000000586c0000,0x00000000587c0000)]
  0x0000000057921000 JavaThread "qtp1642360923-17-selector-ServerConnectorManager@57c6d586/1" [_thread_in_native, id=4232, stack(0x0000000058570000,0x0000000058670000)]
  0x00000000578b5000 JavaThread "qtp1642360923-16-acceptor-0@469f2978-ServerConnector@170acf6e{HTTP/1.1}{0.0.0.0:80}" [_thread_in_native, id=4228, stack(0x0000000058170000,0x0000000058270000)]
  0x0000000056bc0000 JavaThread "Service Thread" daemon [_thread_blocked, id=3604, stack(0x00000000571b0000,0x00000000572b0000)]
  0x0000000056b70800 JavaThread "C1 CompilerThread2" daemon [_thread_blocked, id=3600, stack(0x0000000056fb0000,0x00000000570b0000)]
  0x00000000551dc000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=3596, stack(0x0000000056940000,0x0000000056a40000)]
  0x00000000551d8000 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=3592, stack(0x0000000056a70000,0x0000000056b70000)]
  0x00000000551d3000 JavaThread "Attach Listener" daemon [_thread_blocked, id=3588, stack(0x0000000056840000,0x0000000056940000)]
  0x00000000551cf800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3584, stack(0x00000000566c0000,0x00000000567c0000)]
  0x00000000551b8800 JavaThread "Finalizer" daemon [_thread_blocked, id=3560, stack(0x0000000056500000,0x0000000056600000)]
  0x0000000055171800 JavaThread "Reference Handler" daemon [_thread_blocked, id=3544, stack(0x00000000562a0000,0x00000000563a0000)]

Other Threads:
  0x0000000055169000 VMThread [stack: 0x0000000056110000,0x0000000056210000] [id=3424]
  0x0000000056c0b000 WatcherThread [stack: 0x0000000057330000,0x0000000057430000] [id=3608]

VM state:synchronizing (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x000000000224ba60] Safepoint_lock - owner thread: 0x0000000055169000
[0x000000000224bae0] Threads_lock - owner thread: 0x0000000055169000

Heap:
 PSYoungGen      total 184832K, used 96482K [0x00000000eab00000, 0x00000000fe980000, 0x0000000100000000)
  eden space 179200K, 53% used [0x00000000eab00000,0x00000000f0918ba0,0x00000000f5a00000)
  from space 5632K, 2% used [0x00000000fde80000,0x00000000fdea0000,0x00000000fe400000)
  to   space 5120K, 0% used [0x00000000fe480000,0x00000000fe480000,0x00000000fe980000)
 ParOldGen       total 47104K, used 29727K [0x00000000c0000000, 0x00000000c2e00000, 0x00000000eab00000)
  object space 47104K, 63% used [0x00000000c0000000,0x00000000c1d07dd0,0x00000000c2e00000)
 Metaspace       used 45558K, capacity 46722K, committed 46976K, reserved 1089536K
  class space    used 5649K, capacity 5903K, committed 6016K, reserved 1048576K

Card table byte_map: [0x0000000011d50000,0x0000000011f60000] byte_map_base: 0x0000000011750000

Marking Bits: (ParMarkBitMap*) 0x0000000074db0600
 Begin Bits: [0x00000000125e0000, 0x00000000135e0000)
 End Bits:   [0x00000000135e0000, 0x00000000145e0000)

Polling page: 0x0000000000130000

CodeCache: size=245760Kb used=22745Kb max_used=22762Kb free=223014Kb
 bounds [0x0000000002990000, 0x0000000003ff0000, 0x0000000011990000]
 total_blobs=7274 nmethods=6679 adapters=508
 compilation: enabled

Compilation events (10 events):
Event: 159302.562 Thread 0x00000000551d8000 8439       4       java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask::compareTo (101 bytes)
Event: 159302.563 Thread 0x00000000551d8000 nmethod 8439 0x0000000003fcc8d0 code [0x0000000003fcca20, 0x0000000003fccaf8]
Event: 159602.700 Thread 0x0000000056b70800 8440       3       org.eclipse.jetty.server.session.AbstractSession::isValid (13 bytes)
Event: 159602.700 Thread 0x0000000056b70800 nmethod 8440 0x0000000002b2e690 code [0x0000000002b2e7e0, 0x0000000002b2e990]
Event: 159662.727 Thread 0x0000000056b70800 8441       1       com.vaadin.ui.UI::getUI (2 bytes)
Event: 159662.727 Thread 0x0000000056b70800 nmethod 8441 0x0000000003fcb690 code [0x0000000003fcb7e0, 0x0000000003fcb8f0]
Event: 161433.536 Thread 0x00000000551dc000 8442   !   4       org.eclipse.jetty.server.session.HashSessionManager$Scavenger::run (126 bytes)
Event: 161433.536 Thread 0x00000000551d8000 8443       4       org.eclipse.jetty.util.log.StdErrLog::debug (61 bytes)
Event: 161433.538 Thread 0x00000000551d8000 nmethod 8443 0x0000000003664910 code [0x0000000003664a40, 0x0000000003664ab8]
Event: 161433.594 Thread 0x00000000551dc000 nmethod 8442 0x0000000003fe7dd0 code [0x0000000003fe8000, 0x0000000003fe94b0]

GC Heap History (10 events):
Event: 134999.764 GC heap before
{Heap before GC invocations=79 (full 4):
 PSYoungGen      total 226304K, used 225902K [0x00000000eab00000, 0x00000000ff480000, 0x0000000100000000)
  eden space 225280K, 100% used [0x00000000eab00000,0x00000000f8700000,0x00000000f8700000)
  from space 1024K, 60% used [0x00000000ff380000,0x00000000ff41b8c0,0x00000000ff480000)
  to   space 6656K, 0% used [0x00000000fe780000,0x00000000fe780000,0x00000000fee00000)
 ParOldGen       total 47104K, used 29543K [0x00000000c0000000, 0x00000000c2e00000, 0x00000000eab00000)
  object space 47104K, 62% used [0x00000000c0000000,0x00000000c1cd9db0,0x00000000c2e00000)
 Metaspace       used 45547K, capacity 46722K, committed 46976K, reserved 1089536K
  class space    used 5649K, capacity 5903K, committed 6016K, reserved 1048576K
Event: 134999.768 GC heap after
Heap after GC invocations=79 (full 4):
 PSYoungGen      total 221696K, used 128K [0x00000000eab00000, 0x00000000ff400000, 0x0000000100000000)
  eden space 215040K, 0% used [0x00000000eab00000,0x00000000eab00000,0x00000000f7d00000)
  from space 6656K, 1% used [0x00000000fe780000,0x00000000fe7a0000,0x00000000fee00000)
  to   space 6144K, 0% used [0x00000000fee00000,0x00000000fee00000,0x00000000ff400000)
 ParOldGen       total 47104K, used 29711K [0x00000000c0000000, 0x00000000c2e00000, 0x00000000eab00000)
  object space 47104K, 63% used [0x00000000c0000000,0x00000000c1d03dd0,0x00000000c2e00000)
 Metaspace       used 45547K, capacity 46722K, committed 46976K, reserved 1089536K
  class space    used 5649K, capacity 5903K, committed 6016K, reserved 1048576K
}
Event: 145444.534 GC heap before
{Heap before GC invocations=80 (full 4):
 PSYoungGen      total 221696K, used 215168K [0x00000000eab00000, 0x00000000ff400000, 0x0000000100000000)
  eden space 215040K, 100% used [0x00000000eab00000,0x00000000f7d00000,0x00000000f7d00000)
  from space 6656K, 1% used [0x00000000fe780000,0x00000000fe7a0000,0x00000000fee00000)
  to   space 6144K, 0% used [0x00000000fee00000,0x00000000fee00000,0x00000000ff400000)
 ParOldGen       total 47104K, used 29711K [0x00000000c0000000, 0x00000000c2e00000, 0x00000000eab00000)
  object space 47104K, 63% used [0x00000000c0000000,0x00000000c1d03dd0,0x00000000c2e00000)
 Metaspace       used 45547K, capacity 46722K, committed 46976K, reserved 1089536K
  class space    used 5649K, capacity 5903K, committed 6016K, reserved 1048576K
Event: 145444.538 GC heap after
Heap after GC invocations=80 (full 4):
 PSYoungGen      total 205824K, used 128K [0x00000000eab00000, 0x00000000fee80000, 0x0000000100000000)
  eden space 205312K, 0% used [0x00000000eab00000,0x00000000eab00000,0x00000000f7380000)
  from space 512K, 25% used [0x00000000fee00000,0x00000000fee20000,0x00000000fee80000)
  to   space 6144K, 0% used [0x00000000fe280000,0x00000000fe280000,0x00000000fe880000)
 ParOldGen       total 47104K, used 29711K [0x00000000c0000000, 0x00000000c2e00000, 0x00000000eab00000)
  object space 47104K, 63% used [0x00000000c0000000,0x00000000c1d03dd0,0x00000000c2e00000)
 Metaspace       used 45547K, capacity 46722K, committed 46976K, reserved 1089536K
  class space    used 5649K, capacity 5903K, committed 6016K, reserved 1048576K
}
Event: 154478.429 GC heap before
{Heap before GC invocations=81 (full 4):
 PSYoungGen      total 205824K, used 205440K [0x00000000eab00000, 0x00000000fee80000, 0x0000000100000000)
  eden space 205312K, 100% used [0x00000000eab00000,0x00000000f7380000,0x00000000f7380000)
  from space 512K, 25% used [0x00000000fee00000,0x00000000fee20000,0x00000000fee80000)
  to   space 6144K, 0% used [0x00000000fe280000,0x00000000fe280000,0x00000000fe880000)
 ParOldGen       total 47104K, used 29711K [0x00000000c0000000, 0x00000000c2e00000, 0x00000000eab00000)
  object space 47104K, 63% used [0x00000000c0000000,0x00000000c1d03dd0,0x00000000c2e00000)
 Metaspace       used 45551K, capacity 46722K, committed 46976K, reserved 1089536K
  class space    used 5649K, capacity 5903K, committed 6016K, reserved 1048576K
Event: 154478.432 GC heap after
Heap after GC invocations=81 (full 4):
 PSYoungGen      total 202240K, used 160K [0x00000000eab00000, 0x00000000fee80000, 0x0000000100000000)
  eden space 196096K, 0% used [0x00000000eab00000,0x00000000eab00000,0x00000000f6a80000)
  from space 6144K, 2% used [0x00000000fe280000,0x00000000fe2a8000,0x00000000fe880000)
  to   space 5632K, 0% used [0x00000000fe900000,0x00000000fe900000,0x00000000fee80000)
 ParOldGen       total 47104K, used 29719K [0x00000000c0000000, 0x00000000c2e00000, 0x00000000eab00000)
  object space 47104K, 63% used [0x00000000c0000000,0x00000000c1d05dd0,0x00000000c2e00000)
 Metaspace       used 45551K, capacity 46722K, committed 46976K, reserved 1089536K
  class space    used 5649K, capacity 5903K, committed 6016K, reserved 1048576K
}
Event: 164082.782 GC heap before
{Heap before GC invocations=82 (full 4):
 PSYoungGen      total 202240K, used 196256K [0x00000000eab00000, 0x00000000fee80000, 0x0000000100000000)
  eden space 196096K, 100% used [0x00000000eab00000,0x00000000f6a80000,0x00000000f6a80000)
  from space 6144K, 2% used [0x00000000fe280000,0x00000000fe2a8000,0x00000000fe880000)
  to   space 5632K, 0% used [0x00000000fe900000,0x00000000fe900000,0x00000000fee80000)
 ParOldGen       total 47104K, used 29719K [0x00000000c0000000, 0x00000000c2e00000, 0x00000000eab00000)
  object space 47104K, 63% used [0x00000000c0000000,0x00000000c1d05dd0,0x00000000c2e00000)
 Metaspace       used 45558K, capacity 46722K, committed 46976K, reserved 1089536K
  class space    used 5649K, capacity 5903K, committed 6016K, reserved 1048576K
Event: 164082.786 GC heap after
Heap after GC invocations=82 (full 4):
 PSYoungGen      total 187904K, used 128K [0x00000000eab00000, 0x00000000fe980000, 0x0000000100000000)
  eden space 187392K, 0% used [0x00000000eab00000,0x00000000eab00000,0x00000000f6200000)
  from space 512K, 25% used [0x00000000fe900000,0x00000000fe920000,0x00000000fe980000)
  to   space 5632K, 0% used [0x00000000fde80000,0x00000000fde80000,0x00000000fe400000)
 ParOldGen       total 47104K, used 29727K [0x00000000c0000000, 0x00000000c2e00000, 0x00000000eab00000)
  object space 47104K, 63% used [0x00000000c0000000,0x00000000c1d07dd0,0x00000000c2e00000)
 Metaspace       used 45558K, capacity 46722K, committed 46976K, reserved 1089536K
  class space    used 5649K, capacity 5903K, committed 6016K, reserved 1048576K
}
Event: 173778.650 GC heap before
{Heap before GC invocations=83 (full 4):
 PSYoungGen      total 187904K, used 187520K [0x00000000eab00000, 0x00000000fe980000, 0x0000000100000000)
  eden space 187392K, 100% used [0x00000000eab00000,0x00000000f6200000,0x00000000f6200000)
  from space 512K, 25% used [0x00000000fe900000,0x00000000fe920000,0x00000000fe980000)
  to   space 5632K, 0% used [0x00000000fde80000,0x00000000fde80000,0x00000000fe400000)
 ParOldGen       total 47104K, used 29727K [0x00000000c0000000, 0x00000000c2e00000, 0x00000000eab00000)
  object space 47104K, 63% used [0x00000000c0000000,0x00000000c1d07dd0,0x00000000c2e00000)
 Metaspace       used 45558K, capacity 46722K, committed 46976K, reserved 1089536K
  class space    used 5649K, capacity 5903K, committed 6016K, reserved 1048576K
Event: 173778.653 GC heap after
Heap after GC invocations=83 (full 4):
 PSYoungGen      total 184832K, used 128K [0x00000000eab00000, 0x00000000fe980000, 0x0000000100000000)
  eden space 179200K, 0% used [0x00000000eab00000,0x00000000eab00000,0x00000000f5a00000)
  from space 5632K, 2% used [0x00000000fde80000,0x00000000fdea0000,0x00000000fe400000)
  to   space 5120K, 0% used [0x00000000fe480000,0x00000000fe480000,0x00000000fe980000)
 ParOldGen       total 47104K, used 29727K [0x00000000c0000000, 0x00000000c2e00000, 0x00000000eab00000)
  object space 47104K, 63% used [0x00000000c0000000,0x00000000c1d07dd0,0x00000000c2e00000)
 Metaspace       used 45558K, capacity 46722K, committed 46976K, reserved 1089536K
  class space    used 5649K, capacity 5903K, committed 6016K, reserved 1048576K
}

Deoptimization events (10 events):
Event: 102224.612 Thread 0x0000000056e90000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00000000035bb46c method=java.util.Hashtable.containsKey(Ljava/lang/Object;)Z @ 6
Event: 102224.612 Thread 0x0000000056e90000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x0000000003480724 method=java.util.Hashtable.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; @ 60
Event: 102224.612 Thread 0x0000000056e90000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00000000035bb46c method=java.util.Hashtable.containsKey(Ljava/lang/Object;)Z @ 6
Event: 102224.612 Thread 0x0000000056e90000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x0000000003480724 method=java.util.Hashtable.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; @ 60
Event: 102224.793 Thread 0x0000000056e90000 Uncommon trap: reason=class_check action=maybe_recompile pc=0x0000000003d478a8 method=java.util.ComparableTimSort.binarySort([Ljava/lang/Object;III)V @ 40
Event: 103137.632 Thread 0x0000000056e8e800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00000000039832f8 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 6
Event: 104277.211 Thread 0x0000000056e8e800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x0000000003eafe5c method=java.lang.Character.isISOControl(I)Z @ 16
Event: 106660.954 Thread 0x0000000056e8b800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x0000000003a776bc method=java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object; @ 192
Event: 106660.954 Thread 0x0000000056e8b800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x0000000003a776bc method=java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object; @ 192
Event: 117408.235 Thread 0x0000000056bb9000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x000000000342c264 method=java.lang.ref.SoftReference.get()Ljava/lang/Object; @ 6

Internal exceptions (10 events):
Event: 117407.200 Thread 0x0000000066296800 Exception <a 'java/security/PrivilegedActionException'> (0x00000000ef825f78) thrown at [D:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u77\6540\hotspot\src\share\vm\prims\jvm.cpp, line 1386]
Event: 117407.200 Thread 0x000000006c387000 Exception <a 'java/security/PrivilegedActionException'> (0x00000000ef7e09f8) thrown at [D:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u77\6540\hotspot\src\share\vm\prims\jvm.cpp, line 1386]
Event: 117408.235 Thread 0x0000000056bb9000 Implicit null exception at 0x000000000342bcf3 to 0x000000000342c245
Event: 117410.969 Thread 0x0000000056bb9000 Exception <a 'java/io/IOException'> (0x00000000efa1e130) thrown at [D:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u77\6540\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 123617.033 Thread 0x0000000056e92000 Exception <a 'java/io/IOException'> (0x00000000f7143420) thrown at [D:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u77\6540\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 127983.590 Thread 0x0000000056e8e800 Exception <a 'java/io/IOException'> (0x00000000ef71f768) thrown at [D:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u77\6540\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 127983.592 Thread 0x0000000056e92000 Exception <a 'java/io/IOException'> (0x00000000ef5193e8) thrown at [D:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u77\6540\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 158161.078 Thread 0x0000000056e92000 Exception <a 'java/io/IOException'> (0x00000000ef4f6008) thrown at [D:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u77\6540\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 158354.151 Thread 0x0000000056e8e800 Exception <a 'java/io/IOException'> (0x00000000eee3b798) thrown at [D:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u77\6540\hotspot\src\share\vm\prims\jni.cpp, line 709]
Event: 163436.192 Thread 0x0000000056e92000 Exception <a 'java/io/IOException'> (0x00000000f0e79238) thrown at [D:\re\workspace\8-2-build-windows-amd64-cygwin\jdk8u77\6540\hotspot\src\share\vm\prims\jni.cpp, line 709]

Events (10 events):
Event: 178249.198 Thread 0x00000000699ec800 Thread added: 0x00000000699ec800
Event: 178249.432 Executing VM operation: RevokeBias
Event: 178249.432 Thread 0x00000000699ee800 Thread added: 0x00000000699ee800
Event: 178249.433 Executing VM operation: RevokeBias done
Event: 178249.433 Thread 0x0000000066294000 Thread exited: 0x0000000066294000
Event: 178279.212 Executing VM operation: RevokeBias
Event: 178279.212 Executing VM operation: RevokeBias done
Event: 178279.212 Thread 0x00000000699ed000 Thread added: 0x00000000699ed000
Event: 178279.212 Thread 0x00000000699ec800 Thread exited: 0x00000000699ec800
Event: 178279.446 Executing VM operation: RevokeBias

I found that this could be memory-related and should check for the heap. This seems to be 53% full in this log, but I also found logs with 10% and 99%. Also the stacktraces were always different in those logs.

Somewhere below that I found another (older?) heap before GC, which actually states 100% used. Does that mean it is memory related and increasing the memory for jetty should fix this?

Event: 173778.650 GC heap before
{Heap before GC invocations=83 (full 4):
 PSYoungGen      total 187904K, used 187520K [0x00000000eab00000, 0x00000000fe980000, 0x0000000100000000)
  eden space 187392K, 100% used [0x00000000eab00000,0x00000000f6200000,0x00000000f6200000)
  from space 512K, 25% used [0x00000000fe900000,0x00000000fe920000,0x00000000fe980000)
  to   space 5632K, 0% used [0x00000000fde80000,0x00000000fde80000,0x00000000fe400000)

Update of things I tried:
So far I have checked the memory without any problems. I updated the JRE to 8.0_92. I set the parameters -Xms1g -Xmx2g.

It doesnt seem to resolve the problem though, since I got another crash 3 seconds after running the programm this time: http://pastebin.com/jjEVm09V

Tigerware
  • 3,196
  • 2
  • 23
  • 39
  • It could also be memory-related, in that you might have failing memory; you can test it with a tool like [MemTest86](http://www.memtest86.com/). – Elliott Frisch May 01 '16 at 03:15
  • Memory leak hogging all memory allocated to the program? Have you tried increasing the amount with -xmx and -xms parameters? – user2494817 May 01 '16 at 03:16
  • @ElliottFrisch It has actually happened twice in development on another machine, which is why I think bad memory is unlikely. – Tigerware May 01 '16 at 03:30
  • @user2494817 I havent increased the memory with those parameters yet, since it would be difficult for me to tell if that actually fixes the issue. It just happens randomly every once in a while (maybe 2 weeks) and isnt reproducible otherwise. If you tell me thats its very likely because of low memory, then I will try that. – Tigerware May 01 '16 at 03:32
  • @BluE Hard to say without proper monitoring on the program. I would suggest you take a look at this post: http://stackoverflow.com/questions/3440600/tomcat-jvm-crash-exception-access-violation-0xc0000005?rq=1 It could just be an actual problem within java running for that long. I do see in this dump that the memory violation occurs at HashSet.add method but your memory is low so it could be object corruption or heap corruption occurring. – user2494817 May 01 '16 at 03:36
  • @user2494817 So far the memory violation always occured somewhere else. If it helps I can post another time it happend. I dont know what object/heap corruption is, so I'll look into that. (The comment regarding corruption in the linked question is hard to understand for me) – Tigerware May 01 '16 at 03:51
  • @BluE Well in short; it is a problem with the JVM and using jetty it is in turn unavoidable, but i would suggest trying to up the memory and just see if that helps... Also link me to some of your other error logs on pastebin. – user2494817 May 01 '16 at 03:58
  • Things you could try: a) A different garbage collector b) turning off C1 or C2. Or run interpreter-only on some test machine. And update to _92 while you're at it. Also, this is on windows, does it happen on other operating systems too? – the8472 May 02 '16 at 13:29
  • @user2494817 here is another crash that happend yesterday: http://pastebin.com/jjEVm09V I had updated to _92 and set -Xms1g -Xmx2g. I also checked the memory and it couldnt find anything broken there. The crash happened right after starting the application this time, so it doesnt seem to have anything to do with memory or GC. (The next run was fine though) – Tigerware May 04 '16 at 18:17
  • 1
    @BluE This log shows 10 internal exceptions which might of caused it (or maybe not.) but it is clear that it did crash because of running of native code within the VM (It was a memory access violation and the stack pointer points to a native function). Report this error to jetty they might have more luck with it... – user2494817 May 05 '16 at 03:16

0 Answers0