Tomcat crashing due to native code in JDK: ZIP_GetEntry2

Question

The tomcat server in production crashes often while loggin an error with hs_err_pid below. Logback and log4j both give the same issue hence the problem does not lies with them.

Also, the error comes from Java native code for opening zip files.

Problematic frame:

C [libzip.so+0x12112] ZIP_GetEntry2+0x82

Attached text is the full hs_err_pid file printed by JDK:

---------------  T H R E A D  ---------------

Current thread (0x00007f3cd40a2000):  JavaThread "http-nio-8080-exec-8" daemon [_thread_in_native, id=28088, stack(0x00007f3c024d0000,0x00007f3c025d1000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000000000

[error occurred during error reporting (printing register info), id 0xb]

Stack: [0x00007f3c024d0000,0x00007f3c025d1000],  sp=0x00007f3c025cbb80,  free space=1006k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libzip.so+0x12112]  ZIP_GetEntry2+0x82
C  [libzip.so+0x3ec0]  Java_java_util_zip_ZipFile_getEntry+0xf0
J 364  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007f3ce916c54e [0x00007f3ce916c480+0xce]
J 811 C2 java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (86 bytes) @ 0x00007f3ce9434c68 [0x00007f3ce9434900+0x368]
J 18467 C2 sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (85 bytes) @ 0x00007f3ceb24ed08 [0x00007f3ceb24ec80+0x88]
J 18841 C2 sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (83 bytes) @ 0x00007f3cec3895a0 [0x00007f3cec389480+0x120]
J 55626 C1 java.net.URLClassLoader$1.run()Ljava/lang/Class; (63 bytes) @ 0x00007f3cebd1b92c [0x00007f3cebd1b720+0x20c]
J 55248 C1 java.net.URLClassLoader$1.run()Ljava/lang/Object; (5 bytes) @ 0x00007f3ced5fc4fc [0x00007f3ced5fc480+0x7c]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x690dd6]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x72a09c]  JVM_DoPrivileged+0x27c
J 752  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (0 bytes) @ 0x00007f3ce9250c55 [0x00007f3ce9250b80+0xd5]
J 7297 C2 java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class; (47 bytes) @ 0x00007f3cea70bcbc [0x00007f3cea70bc20+0x9c]
J 7891 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122 bytes) @ 0x00007f3ceaa307f0 [0x00007f3ceaa304a0+0x350]
J 10498 C2 sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (85 bytes) @ 0x00007f3ceaf1e590 [0x00007f3ceaf1e400+0x190]
J 7891 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122 bytes) @ 0x00007f3ceaa30740 [0x00007f3ceaa304a0+0x2a0]
J 9654 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (93 bytes) @ 0x00007f3cea1de078 [0x00007f3cea1ddfc0+0xb8]
v  ~StubRoutines::call_stub
V  [libjvm.so+0x690dd6]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x6912e1]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V  [libjvm.so+0x691736]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Handle, Thread*)+0x56
V  [libjvm.so+0xa3c7a0]  SystemDictionary::load_instance_class(Symbol*, Handle, Thread*)+0x3f0
V  [libjvm.so+0xa3b6ec]  SystemDictionary::resolve_instance_class_or_null(Symbol*, Handle, Handle, Thread*)+0x78c
V  [libjvm.so+0xa3cbc3]  SystemDictionary::resolve_or_fail(Symbol*, Handle, Handle, bool, Thread*)+0x33
V  [libjvm.so+0x70f46e]  find_class_from_class_loader(JNIEnv_*, Symbol*, unsigned char, Handle, Handle, unsigned char, Thread*)+0x3e
V  [libjvm.so+0x716071]  JVM_FindClassFromCaller+0x2e1
C  [libjava.so+0xe320]  Java_java_lang_Class_forName0+0x130
J 1597  java.lang.Class.forName0(Ljava/lang/String;ZLjava/lang/ClassLoader;Ljava/lang/Class;)Ljava/lang/Class; (0 bytes) @ 0x00007f3ce945f9b4 [0x00007f3ce945f8c0+0xf4]
J 8069 C2 java.lang.Class.forName(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class; (53 bytes) @ 0x00007f3ceab0de88 [0x00007f3ceab0de60+0x28]
j  org.apache.catalina.loader.WebappClassLoaderBase.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+653
J 49737 C2 org.apache.logging.log4j.core.impl.ThrowableProxy.loadClass(Ljava/lang/ClassLoader;Ljava/lang/String;)Ljava/lang/Class; (43 bytes) @ 0x00007f3ceb7d5da4 [0x00007f3ceb7d5b60+0x244]
J 50915 C2 org.apache.logging.log4j.core.impl.ThrowableProxy.toExtendedStackTrace(Ljava/util/Stack;Ljava/util/Map;[Ljava/lang/StackTraceElement;[Ljava/lang/StackTraceElement;)[Lorg/apache/logging/log4j/core/impl/ExtendedStackTraceElement; (333 bytes) @ 0x00007f3cefbebcfc [0x00007f3cefbeb340+0x9bc]
J 48302 C1 org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(Ljava/lang/Throwable;Ljava/util/Set;)V (121 bytes) @ 0x00007f3ceeae9f84 [0x00007f3ceeae96c0+0x8c4]
J 52797 C1 org.apache.logging.log4j.core.impl.Log4jLogEvent.getThrownProxy()Lorg/apache/logging/log4j/core/impl/ThrowableProxy; (34 bytes) @ 0x00007f3cee06e06c [0x00007f3cee06dec0+0x1ac]
J 52796 C1 org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter.format(Lorg/apache/logging/log4j/core/LogEvent;Ljava/lang/StringBuilder;)V (224 bytes) @ 0x00007f3cf07eb824 [0x00007f3cf07eb700+0x124]
J 55082 C2 org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(Lorg/apache/logging/log4j/core/LogEvent;Ljava/lang/StringBuilder;)Ljava/lang/StringBuilder; (71 bytes) @ 0x00007f3cf1e927c4 [0x00007f3cf1e92740+0x84]
J 52012 C1 org.apache.logging.log4j.core.layout.PatternLayout.encode(Lorg/apache/logging/log4j/core/LogEvent;Lorg/apache/logging/log4j/core/layout/ByteBufferDestination;)V (53 bytes) @ 0x00007f3cedb9e29c [0x00007f3cedb9df00+0x39c]
J 52011 C1 org.apache.logging.log4j.core.layout.PatternLayout.encode(Ljava/lang/Object;Lorg/apache/logging/log4j/core/layout/ByteBufferDestination;)V (10 bytes) @ 0x00007f3cee84a074 [0x00007f3cee849e80+0x1f4]
J 52010 C1 org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Lorg/apache/logging/log4j/core/LogEvent;)V (38 bytes) @ 0x00007f3cee84a6bc [0x00007f3cee84a500+0x1bc]
J 52008 C1 org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(Lorg/apache/logging/log4j/core/LogEvent;)V (62 bytes) @ 0x00007f3ceda80504 [0x00007f3ceda803a0+0x164]
J 51194 C1 org.apache.logging.log4j.core.appender.RollingFileAppender.append(Lorg/apache/logging/log4j/core/LogEvent;)V (17 bytes) @ 0x00007f3cee493234 [0x00007f3cee492dc0+0x474]
J 51193 C1 org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(Lorg/apache/logging/log4j/core/LogEvent;)V (36 bytes) @ 0x00007f3cec6aded4 [0x00007f3cec6addc0+0x114]
J 51189 C1 org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Lorg/apache/logging/log4j/core/LogEvent;)V (36 bytes) @ 0x00007f3ceff82514 [0x00007f3ceff821c0+0x354]
J 50719 C1 org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(Lorg/apache/logging/log4j/core/LogEvent;)V (30 bytes) @ 0x00007f3cedf6081c [0x00007f3cedf60060+0x7bc]
J 49854 C1 org.apache.logging.log4j.core.config.LoggerConfig.log(Ljava/lang/String;Ljava/lang/String;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (220 bytes) @ 0x00007f3cee9fd6fc [0x00007f3cee9fb700+0x1ffc]
J 49020 C1 org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(Lorg/apache/logging/log4j/util/Supplier;Ljava/lang/String;Ljava/lang/String;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (51 bytes) @ 0x00007f3cead3403c [0x00007f3cead33a80+0x5bc]
J 47668 C1 org.apache.logging.log4j.core.Logger.logMessage(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (53 bytes) @ 0x00007f3cef1bb7c4 [0x00007f3cef1bb3e0+0x3e4]
J 49019 C1 org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (30 bytes) @ 0x00007f3cef5f6b4c [0x00007f3cef5f6a40+0x10c]
J 47674 C1 org.apache.logging.slf4j.Log4jLogger.error(Ljava/lang/String;Ljava/lang/Throwable;)V (19 bytes) @ 0x00007f3cf10658cc [0x00007f3cf1065480+0x44c]
J 43454 C2 com.nm.impl.service.catalog.CatalogServiceImpl.decorateProductDetailWithAlternative(Ljava/util/List;)Ljava/util/List; (203 bytes) @ 0x00007f3cf0657130 [0x00007f3cf064db00+0x9630]
J 54962 C2 com.nm.impl.service.catalog.CatalogServiceImpl.getCatalogResults(Lcom/nm/query/CatalogQuery;)Lcom/nm/rest/response/catalog/CatalogResults; (1136 bytes) @ 0x00007f3cf1e1cc80 [0x00007f3cf1e19360+0x3920]
J 51398 C2 com.nm.impl.service.catalog.CatalogBrowseServiceImpl.getCatalogResults(Lcom/nm/rest/request/catalog/CatalogServeRequest;)Lcom/nm/rest/response/catalog/CatalogResults; (1331 bytes) @ 0x00007f3cf1845a34 [0x00007f3cf1845040+0x9f4]
J 49137 C2 com.nm.rest.resource.catalog.CatalogResource.serveWebCatalog(Ljava/lang/Long;SZLjava/lang/Double;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;ZLjava/lang/Long;Ljava/lang/Long;Ljava/lang/String;Ljava/lang/String;IILjava/lang/String;Ljava/lang/String;Ljava/lang/String;ZLjava/lang/String;)Lcom/nm/rest/response/catalog/CatalogResults; (188 bytes) @ 0x00007f3cef79e4e4 [0x00007f3cef79de60+0x684]
J 49140 C2 sun.reflect.GeneratedMethodAccessor1238.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (466 bytes) @ 0x00007f3cf06aee64 [0x00007f3cf06ae9e0+0x484]
J 45621 C2 org.jboss.resteasy.core.MethodInjectorImpl.invoke(Lorg/jboss/resteasy/spi/HttpRequest;Lorg/jboss/resteasy/spi/HttpResponse;Ljava/lang/Object;)Ljava/lang/Object; (431 bytes) @ 0x00007f3cebf477d8 [0x00007f3cebf47400+0x3d8]
J 53007 C2 org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(Lorg/jboss/resteasy/spi/HttpRequest;Lorg/jboss/resteasy/spi/HttpResponse;Ljava/lang/Object;)Lorg/jboss/resteasy/specimpl/BuiltResponse; (539 bytes) @ 0x00007f3ced0bdd0c [0x00007f3ced0bdc60+0xac]
J 36880 C2 org.jboss.resteasy.core.SynchronousDispatcher.invoke(Lorg/jboss/resteasy/spi/HttpRequest;Lorg/jboss/resteasy/spi/HttpResponse;Lorg/jboss/resteasy/core/ResourceInvoker;)V (74 bytes) @ 0x00007f3cef6b0404 [0x00007f3cef6afe00+0x604]
J 47512 C2 org.jboss.resteasy.core.SynchronousDispatcher.invoke(Lorg/jboss/resteasy/spi/HttpRequest;Lorg/jboss/resteasy/spi/HttpResponse;)V (70 bytes) @ 0x00007f3cf10f3698 [0x00007f3cf10f32c0+0x3d8]
J 51296 C2 org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (13 bytes) @ 0x00007f3cf17b5ab4 [0x00007f3cf17b50c0+0x9f4]
J 35311 C2 javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (352 bytes) @ 0x00007f3cee3b1f18 [0x00007f3cee3b1900+0x618]
J 30287 C2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (574 bytes) @ 0x00007f3cee6b9404 [0x00007f3cee6b92e0+0x124]
J 35575 C2 org.apache.tomcat.websocket.server.WsFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (472 bytes) @ 0x00007f3cef33ff0c [0x00007f3cef33f920+0x5ec]
J 30287 C2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (574 bytes) @ 0x00007f3cee6b95a4 [0x00007f3cee6b92e0+0x2c4]
J 37440 C2 org.apache.catalina.filters.CorsFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (496 bytes) @ 0x00007f3cef776d98 [0x00007f3cef775b40+0x1258]
J 30287 C2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (574 bytes) @ 0x00007f3cee6b95a4 [0x00007f3cee6b92e0+0x2c4]
J 53290 C2 com.nm.security.ApiKeyFilter.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;Ljavax/servlet/FilterChain;)V (706 bytes) @ 0x00007f3cf1acc644 [0x00007f3cf1aca740+0x1f04]
J 30287 C2 org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V (574 bytes) @ 0x00007f3cee6b95a4 [0x00007f3cee6b92e0+0x2c4]
J 35566 C2 org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (1414 bytes) @ 0x00007f3cef335e68 [0x00007f3cef335c40+0x228]
J 51281 C2 org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (172 bytes) @ 0x00007f3cf176b280 [0x00007f3cf176a8e0+0x9a0]
J 51284 C2 org.apache.catalina.authenticator.AuthenticatorBase.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (903 bytes) @ 0x00007f3ceda35728 [0x00007f3ceda35240+0x4e8]
J 45521 C2 org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V (379 bytes) @ 0x00007f3cee50e5cc [0x00007f3cee50e1e0+0x3ec]
J 45522 C2 org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V (857 bytes) @ 0x00007f3cee50731c [0x00007f3cee506d00+0x61c]
J 49415 C2 org.apache.coyote.http11.AbstractHttp11Processor.process(Lorg/apache/tomcat/util/net/SocketWrapper;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (1153 bytes) @ 0x00007f3cf156d638 [0x00007f3cf156cfa0+0x698]
J 47161 C2 org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Lorg/apache/tomcat/util/net/SocketWrapper;Lorg/apache/tomcat/util/net/SocketStatus;)Lorg/apache/tomcat/util/net/AbstractEndpoint$Handler$SocketState; (814 bytes) @ 0x00007f3cf0f077f0 [0x00007f3cf0f07280+0x570]
J 47159 C2 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(Ljava/nio/channels/SelectionKey;Lorg/apache/tomcat/util/net/NioEndpoint$KeyAttachment;)V (785 bytes) @ 0x00007f3cf0f0fdb4 [0x00007f3cf0f0fce0+0xd4]
J 34980 C2 org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run()V (106 bytes) @ 0x00007f3cecd97b68 [0x00007f3cecd979c0+0x1a8]
J 43089% C2 java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V (225 bytes) @ 0x00007f3cee701548 [0x00007f3cee7011c0+0x388]
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j  org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run()V+4
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
V  [libjvm.so+0x690dd6]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1056
V  [libjvm.so+0x6912e1]  JavaCalls::call_virtual(JavaValue*, KlassHandle, Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x321
V  [libjvm.so+0x691787]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, Symbol*, Symbol*, Thread*)+0x47
V  [libjvm.so+0x72cb00]  thread_entry(JavaThread*, Thread*)+0xa0
V  [libjvm.so+0xa75543]  JavaThread::thread_main_inner()+0x103
V  [libjvm.so+0xa7568c]  JavaThread::run()+0x11c
V  [libjvm.so+0x926268]  java_start(Thread*)+0x108
C  [libpthread.so.0+0x7a51]  start_thread+0xd1

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 364  java.util.zip.ZipFile.getEntry(J[BZ)J (0 bytes) @ 0x00007f3ce916c4d8 [0x00007f3ce916c480+0x58]
J 811 C2 java.util.zip.ZipFile.getEntry(Ljava/lang/String;)Ljava/util/zip/ZipEntry; (86 bytes) @ 0x00007f3ce9434c68 [0x00007f3ce9434900+0x368]
J 18467 C2 sun.misc.URLClassPath$JarLoader.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (85 bytes) @ 0x00007f3ceb24ed08 [0x00007f3ceb24ec80+0x88]
J 18841 C2 sun.misc.URLClassPath.getResource(Ljava/lang/String;Z)Lsun/misc/Resource; (83 bytes) @ 0x00007f3cec3895a0 [0x00007f3cec389480+0x120]
J 55626 C1 java.net.URLClassLoader$1.run()Ljava/lang/Class; (63 bytes) @ 0x00007f3cebd1b92c [0x00007f3cebd1b720+0x20c]
J 55248 C1 java.net.URLClassLoader$1.run()Ljava/lang/Object; (5 bytes) @ 0x00007f3ced5fc4fc [0x00007f3ced5fc480+0x7c]
v  ~StubRoutines::call_stub
J 752  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedExceptionAction;Ljava/security/AccessControlContext;)Ljava/lang/Object; (0 bytes) @ 0x00007f3ce9250be3 [0x00007f3ce9250b80+0x63]
J 7297 C2 java.net.URLClassLoader.findClass(Ljava/lang/String;)Ljava/lang/Class; (47 bytes) @ 0x00007f3cea70bcbc [0x00007f3cea70bc20+0x9c]
J 7891 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122 bytes) @ 0x00007f3ceaa307f0 [0x00007f3ceaa304a0+0x350]
J 10498 C2 sun.misc.Launcher$AppClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (85 bytes) @ 0x00007f3ceaf1e590 [0x00007f3ceaf1e400+0x190]
J 7891 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122 bytes) @ 0x00007f3ceaa30740 [0x00007f3ceaa304a0+0x2a0]
J 9654 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (93 bytes) @ 0x00007f3cea1de078 [0x00007f3cea1ddfc0+0xb8]
v  ~StubRoutines::call_stub
J 1597  java.lang.Class.forName0(Ljava/lang/String;ZLjava/lang/ClassLoader;Ljava/lang/Class;)Ljava/lang/Class; (0 bytes) @ 0x00007f3ce945f93a [0x00007f3ce945f8c0+0x7a]
J 8069 C2 java.lang.Class.forName(Ljava/lang/String;ZLjava/lang/ClassLoader;)Ljava/lang/Class; (53 bytes) @ 0x00007f3ceab0de88 [0x00007f3ceab0de60+0x28]
j  org.apache.catalina.loader.WebappClassLoaderBase.loadClass(Ljava/lang/String;Z)Ljava/lang/Class;+653
J 49737 C2 org.apache.logging.log4j.core.impl.ThrowableProxy.loadClass(Ljava/lang/ClassLoader;Ljava/lang/String;)Ljava/lang/Class; (43 bytes) @ 0x00007f3ceb7d5da4 [0x00007f3ceb7d5b60+0x244]
J 50915 C2 org.apache.logging.log4j.core.impl.ThrowableProxy.toExtendedStackTrace(Ljava/util/Stack;Ljava/util/Map;[Ljava/lang/StackTraceElement;[Ljava/lang/StackTraceElement;)[Lorg/apache/logging/log4j/core/impl/ExtendedStackTraceElement; (333 bytes) @ 0x00007f3cefbebcfc [0x00007f3cefbeb340+0x9bc]
J 48302 C1 org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(Ljava/lang/Throwable;Ljava/util/Set;)V (121 bytes) @ 0x00007f3ceeae9f84 [0x00007f3ceeae96c0+0x8c4]
J 52797 C1 org.apache.logging.log4j.core.impl.Log4jLogEvent.getThrownProxy()Lorg/apache/logging/log4j/core/impl/ThrowableProxy; (34 bytes) @ 0x00007f3cee06e06c [0x00007f3cee06dec0+0x1ac]
J 52796 C1 org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter.format(Lorg/apache/logging/log4j/core/LogEvent;Ljava/lang/StringBuilder;)V (224 bytes) @ 0x00007f3cf07eb824 [0x00007f3cf07eb700+0x124]
J 55082 C2 org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(Lorg/apache/logging/log4j/core/LogEvent;Ljava/lang/StringBuilder;)Ljava/lang/StringBuilder; (71 bytes) @ 0x00007f3cf1e927c4 [0x00007f3cf1e92740+0x84]
J 52012 C1 org.apache.logging.log4j.core.layout.PatternLayout.encode(Lorg/apache/logging/log4j/core/LogEvent;Lorg/apache/logging/log4j/core/layout/ByteBufferDestination;)V (53 bytes) @ 0x00007f3cedb9e29c [0x00007f3cedb9df00+0x39c]
J 52011 C1 org.apache.logging.log4j.core.layout.PatternLayout.encode(Ljava/lang/Object;Lorg/apache/logging/log4j/core/layout/ByteBufferDestination;)V (10 bytes) @ 0x00007f3cee84a074 [0x00007f3cee849e80+0x1f4]
J 52010 C1 org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Lorg/apache/logging/log4j/core/LogEvent;)V (38 bytes) @ 0x00007f3cee84a6bc [0x00007f3cee84a500+0x1bc]
J 52008 C1 org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(Lorg/apache/logging/log4j/core/LogEvent;)V (62 bytes) @ 0x00007f3ceda80504 [0x00007f3ceda803a0+0x164]
J 51194 C1 org.apache.logging.log4j.core.appender.RollingFileAppender.append(Lorg/apache/logging/log4j/core/LogEvent;)V (17 bytes) @ 0x00007f3cee493234 [0x00007f3cee492dc0+0x474]
J 51193 C1 org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(Lorg/apache/logging/log4j/core/LogEvent;)V (36 bytes) @ 0x00007f3cec6aded4 [0x00007f3cec6addc0+0x114]
J 51189 C1 org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Lorg/apache/logging/log4j/core/LogEvent;)V (36 bytes) @ 0x00007f3ceff82514 [0x00007f3ceff821c0+0x354]
J 50719 C1 org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(Lorg/apache/logging/log4j/core/LogEvent;)V (30 bytes) @ 0x00007f3cedf6081c [0x00007f3cedf60060+0x7bc]
J 49854 C1 org.apache.logging.log4j.core.config.LoggerConfig.log(Ljava/lang/String;Ljava/lang/String;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (220 bytes) @ 0x00007f3cee9fd6fc [0x00007f3cee9fb700+0x1ffc]
J 49020 C1 org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(Lorg/apache/logging/log4j/util/Supplier;Ljava/lang/String;Ljava/lang/String;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (51 bytes) @ 0x00007f3cead3403c [0x00007f3cead33a80+0x5bc]
J 47668 C1 org.apache.logging.log4j.core.Logger.logMessage(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (53 bytes) @ 0x00007f3cef1bb7c4 [0x00007f3cef1bb3e0+0x3e4]
J 49019 C1 org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(Ljava/lang/String;Lorg/apache/logging/log4j/Level;Lorg/apache/logging/log4j/Marker;Lorg/apache/logging/log4j/message/Message;Ljava/lang/Throwable;)V (30 bytes) @ 0x00007f3cef5f6b4c [0x00007f3cef5f6a40+0x10c]
J 47674 C1 org.apache.logging.slf4j.Log4jLogger.error(Ljava/lang/String;Ljava/lang/Throwable;)V (19 bytes) @ 0x00007f3cf10658cc [0x00007f3cf1065480+0x44c]
J 43454 C2 

Internal exceptions (10 events):
Event: 18089.677 Thread 0x00007f3c6092d000 Exception <a 'java/net/ConnectException'> (0x000000079006e590) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 18089.677 Thread 0x00007f3c6092d000 Exception <a 'java/net/ConnectException'> (0x00000007901ba450) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 18089.677 Thread 0x00007f3c6092d000 Exception <a 'java/net/ConnectException'> (0x00000007901bb1c8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 18089.677 Thread 0x00007f3c6092d000 Exception <a 'java/net/ConnectException'> (0x00000007901bbf78) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 18089.746 Thread 0x00007f3c4401a000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x000000076676fee0) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 18090.057 Thread 0x00007f3cd40a0000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x0000000795626078) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 18090.839 Thread 0x00007f3cd40af000 Exception <a 'java/net/SocketTimeoutException': Read timed out> (0x000000079b1d2d88) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 18091.177 Thread 0x00007f3c60054800 Exception <a 'java/net/SocketException'> (0x00000007206b58e8) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 709]
Event: 18091.177 Thread 0x00007f3c60054800 Exception <a 'java/net/SocketException': Socket closed> (0x00000007206b5de0) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 735]
Event: 18091.310 Thread 0x00007f3cd40a2000 Exception <a 'java/net/SocketException'> (0x00000007214e1750) thrown at [/HUDSON3/workspace/8-2-build-linux-amd64/jdk8u121/8372/hotspot/src/share/vm/prims/jni.cpp, line 709]


VM Arguments:
jvm_args: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/heap -Djava.util.Arrays.useLegacyMergeSort=true -Xms3072M -Xmx7680M -Dsun.zip.disableMemoryMapping=true

Is the ZIP file being overridden while in use? https://bugs.openjdk.java.net/browse/JDK-8160933 — Luciano van der Veekens, Apr 14 '17 at 10:34
No, the zip files are being used by log4j to extract classes, are not overridden in the code. Can you help me with cases where java code is changing the classes within zip file. Also, I checked the attached link, and have added disableMemoryMapping flag in JVM parameters but to no avail. — Ankush, Apr 15 '17 at 06:08

score 1 · Answer 1 · answered Feb 07 '19 at 12:18

The root cause and workaround: https://blogs.oracle.com/poonam/crashes-in-zipgetentry

The workaround is to add this:

-Dsun.zip.disableMemoryMapping=true

The documentation of the issue and it's resolution in JAVA9+: https://bugs.openjdk.java.net/browse/JDK-8145260

score 0 · Answer 2 · answered Oct 16 '17 at 05:47

After some hit and trial, I was able to resolve the problem but could not reach the root cause.

Old ( Problematic ) setup: CentOS 6 + Oracle JDK 8.121.
New environment: CentOS 7 + openjdk 8.144.

Note: openjdk does not have Java FX bundled like Oracle JDK (more here)

In the production cluster we have 3 machines(same code) out of which one has the new setup ( CentOS 7 + openjdk 8.144) while other 2 have ( CentOS 6 + Oracle JDK 8.121). The second setup is not producing this error while the other two are still.

I cannot blame the Oracle JDK as it is built on openjdk internally with some tweaks and bundled JavaFX. The problem is with OS not able to lock the file when someone else has already opened it. Hence, it is the OS which is likely to be the culprit.

Tomcat crashing due to native code in JDK: ZIP_GetEntry2

Problematic frame:

C [libzip.so+0x12112] ZIP_GetEntry2+0x82

2 Answers2