1

We are seeing a JVM running on ignite cluster crashes with seg-fault, the error trace taking us to org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.write0

Below is full stack-trace of error.

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f47f27d563a, pid=40132, tid=0x00007f4716cac700
#
# JRE version: OpenJDK Runtime Environment (8.0_222-b10) (build
1.8.0_222-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.222-b10 mixed mode linux-amd64
compressed oops)
# Problematic frame:
# J 58563 C2
org.apache.ignite.internal.binary.BinaryFieldAccessor$DefaultFinalClassAccessor.write0(Ljava/lang/Object;Lorg/apache/ignite/internal/binary/BinaryWriterExImpl;)V
(761 bytes) @ 0x00007f47f27d563a [0x00007f47f27d5500+0x13a]
#
# Failed to write core dump. Core dumps have been disabled. To enable core
dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

While trying to google around JVM crash reasons I found this https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/crashes001.html From this it look crash category: "5.1.2 Crash in Compiled Code" as we are seeing problematic frame is a java frame : "J"

We are running ignite 2.7.0 with openjdk 1.8 on RHEL machines. We got the basic OS/Hardware level check done by respective team.

Can someone please check and suggest what could be the underlying cause ? Is there any Unsafe/Native API access with this api which might lead to crash or any bug with this version or any openjdk bug for such crashes ?

I can add the full error log file (hs_err_pid) if needed.

Update: Adding the hs_error log file here, Please download from link. hs_err_pid file

Thanks, Tarun

tarunk
  • 549
  • 2
  • 7
  • 17
  • Maybe the "apache" library version doesn't match the java 1.8 version, maybe it is for older java? – Krzysztof Cichocki Jan 30 '20 at 09:32
  • We have been running on this version of java and ignite for quite some time and its not coming always. It goes away with restart and then reappear after few days. – tarunk Jan 30 '20 at 09:42
  • Just in time compilation or such can cause such. And I once encountered a null `this`: an inner serializable class and and outer non-serializable one. So using a **code checker** can be very useful. Unsafe / native classes someone else with RHEL experience must answer. Myself the first thing I do, is scanning for viruses, checking for the disk errors. – Joop Eggen Jan 30 '20 at 09:50
  • Thanks @Joop Eggen for your input. We got the basic level check done at OS/Hardware level by IT peoples. Unfortunately since the problematic frame coming in error trace, is not our code we are not much sure of its internals. – tarunk Jan 30 '20 at 10:11
  • I can see you have two crashers now: this one and https://stackoverflow.com/questions/59835460/how-to-know-the-reason-for-jvm-crashing-with-segfault - did you consider upgrading to 2.7.6, btw? – alamar Jan 30 '20 at 10:37
  • The error file could add more insights, this does not tell much so far. – Eugene Jan 30 '20 at 10:51
  • Hi @Eugene, I have attached the error file, Please download from link given in question as Update. – tarunk Jan 30 '20 at 17:26
  • Hi @alamar Yes we have seen this earlier too but pointing to different Ignite api. Hence raised this separately. We are planning to upgrade but it may take some time to prioritize it. – tarunk Jan 30 '20 at 17:29
  • 404 for the file download – Eugene Jan 30 '20 at 17:45
  • @Eugene Can you please try with below url https://filebin.net/8d71j5tyniwgr4dy – tarunk Jan 31 '20 at 07:11

0 Answers0