0

When I restarted my JBoss app I could see below error,

WARN  [com.arjuna.ats.arjuna] (Periodic Recovery) ARJUNA012087: Activate of atomic action with id 0:ff000ac1af59d:1aff0d21:5dbaaaacd:cbaaba and type /StateManager/BasicAction/TwoPhaseCoordinator/ArjunaTransactionImple/ServerTransaction/JCA unexpectedly failed, could not load state.
FATAL [com.arjuna.ats.jts] (Periodic Recovery) ARJUNA022006: The ORB has not been initialized yet
ERROR [stderr] (Periodic Recovery) Exception in thread "Periodic Recovery" com.arjuna.ats.arjuna.exceptions.FatalError
ERROR [stderr] (Periodic Recovery)  at com.arjuna.ats.internal.jts.ORBManager.getPOA(ORBManager.java:96)
ERROR [stderr] (Periodic Recovery)  at com.arjuna.ats.internal.jts.orbspecific.ControlImple.createTransactionHandle(ControlImple.java:524)
ERROR [stderr] (Periodic Recovery)  at com.arjuna.ats.internal.jts.orbspecific.interposition.ServerControl.<init>(ServerControl.java:223)
ERROR [stderr] (Periodic Recovery)  at com.arjuna.ats.internal.jta.transaction.jts.subordinate.jca.SubordinateAtomicTransaction.<init>(SubordinateAtomicTransaction.java:71)
ERROR [stderr] (Periodic Recovery)  at com.arjuna.ats.internal.jta.transaction.jts.subordinate.jca.TransactionImple.<init>(TransactionImple.java:70)

Jboss started successfully but not in a healthy state. I could not find the root cause of why this issue has happened. Can someone please help to understand what is this error indicate! and what are the possible cause of this error?

Also seen this error during remote ejb call,

@JBOSS@FATAL [com.arjuna.ats.jts] (EJB default - 6) ARJUNA022006: The ORB has not been initialized yet
@JBOSS@ERROR [org.jboss.as.ejb3.invocation] (EJB default - 6) JBAS014134: EJB Invocation failed on component MediationServiceClientBean for method public abstract java.lang.Object com.subhash.Test.testMe(com.subhash.Test) throws com.subhash.Test.TestException: com.arjuna.ats.arjuna.exceptions.FatalError
            at com.arjuna.ats.internal.jts.ORBManager.getPOA(ORBManager.java:96)
            at com.arjuna.ats.internal.jts.orbspecific.ControlImple.createTransactionHandle(ControlImple.java:524)
            at com.arjuna.ats.internal.jts.orbspecific.interposition.ServerControl.<init>(ServerControl.java:223)
            at com.arjuna.ats.internal.jta.transaction.jts.subordinate.jca.SubordinateAtomicTransaction.<init>(SubordinateAtomicTransaction.java:51)
            at com.arjuna.ats.internal.jta.transaction.jts.subordinate.jca.TransactionImple.<init>(TransactionImple.java:57)
            at com.arjuna.ats.internal.jta.transaction.jts.jca.TransactionImporterImple.addImportedTransaction(TransactionImporterImple.java:233)
            at com.arjuna.ats.internal.jta.transaction.jts.jca.TransactionImporterImple.importRemoteTransaction(TransactionImporterImple.java:90)
            at com.arjuna.ats.internal.jta.transaction.jts.jca.TransactionImporterImple.importTransaction(TransactionImporterImple.java:70)
            at org.jboss.as.ejb3.remote.EJBRemoteTransactionsRepository.importTransaction(EJBRemoteTransactionsRepository.java:187) [jboss-as-ejb3-7.5.21.Final-redhat-1-bz-1653367.jar:7.5.21.Final-redhat-SNAPSHOT]
            at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.createOrResumeXidTransaction(EJBRemoteTransactionPropagatingInterceptor.java:126) [jboss-as-ejb3-7.5.21.Final-redhat-1-bz-1653367.jar:7.5.21.Final-redhat-SNAPSHOT]
            at org.jboss.as.ejb3.remote.EJBRemoteTransactionPropagatingInterceptor.processInvocation(EJBRemoteTransactionPropagatingInterceptor.java:72) [jboss-as-ejb3-7.5.21.Final-redhat-1-bz-1653367.jar:7.5.21.Final-redhat-SNAPSHOT]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.3.Final-redhat-1.jar:1.1.3.Final-redhat-1]
            at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3-7.5.21.Final-redhat-1-bz-1653367.jar:7.5.21.Final-redhat-SNAPSHOT]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.3.Final-redhat-1.jar:1.1.3.Final-redhat-1]
            at org.jboss.as.ejb3.component.invocationmetrics.WaitTimeInterceptor.processInvocation(WaitTimeInterceptor.java:43) [jboss-as-ejb3-7.5.21.Final-redhat-1-bz-1653367.jar:7.5.21.Final-redhat-SNAPSHOT]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.3.Final-redhat-1.jar:1.1.3.Final-redhat-1]
            at org.jboss.as.ejb3.security.SecurityContextInterceptor.processInvocation(SecurityContextInterceptor.java:100) [jboss-as-ejb3-7.5.21.Final-redhat-1-bz-1653367.jar:7.5.21.Final-redhat-SNAPSHOT]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.3.Final-redhat-1.jar:1.1.3.Final-redhat-1]
            at org.jboss.as.ejb3.deployment.processors.StartupAwaitInterceptor.processInvocation(StartupAwaitInterceptor.java:22) [jboss-as-ejb3-7.5.21.Final-redhat-1-bz-1653367.jar:7.5.21.Final-redhat-SNAPSHOT]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.3.Final-redhat-1.jar:1.1.3.Final-redhat-1]
            at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3-7.5.21.Final-redhat-1-bz-1653367.jar:7.5.21.Final-redhat-SNAPSHOT]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.3.Final-redhat-1.jar:1.1.3.Final-redhat-1]
            at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3-7.5.21.Final-redhat-1-bz-1653367.jar:7.5.21.Final-redhat-SNAPSHOT]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.3.Final-redhat-1.jar:1.1.3.Final-redhat-1]
            at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee-7.5.21.Final-redhat-1.jar:7.5.21.Final-redhat-1]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.3.Final-redhat-1.jar:1.1.3.Final-redhat-1]
            at org.jboss.as.ejb3.component.interceptors.AdditionalSetupInterceptor.processInvocation(AdditionalSetupInterceptor.java:55) [jboss-as-ejb3-7.5.21.Final-redhat-1-bz-1653367.jar:7.5.21.Final-redhat-SNAPSHOT]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.3.Final-redhat-1.jar:1.1.3.Final-redhat-1]
            at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee-7.5.21.Final-redhat-1.jar:7.5.21.Final-redhat-1]
            at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation-1.1.3.Final-redhat-1.jar:1.1.3.Final-redhat-1]
            at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation-1.1.3.Final-redhat-1.jar:1.1.3.Final-redhat-1]
            at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:189) [jboss-as-ee-7.5.21.Final-redhat-1.jar:7.5.21.Final-redhat-1]
            at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.invokeMethod(MethodInvocationMessageHandler.java:327) [jboss-as-ejb3-7.5.21.Final-redhat-1-bz-1653367.jar:7.5.21.Final-redhat-SNAPSHOT]
            at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler.access$100(MethodInvocationMessageHandler.java:69) [jboss-as-ejb3-7.5.21.Final-redhat-1-bz-1653367.jar:7.5.21.Final-redhat-SNAPSHOT]
            at org.jboss.as.ejb3.remote.protocol.versionone.MethodInvocationMessageHandler$1.run(MethodInvocationMessageHandler.java:202) [jboss-as-ejb3-7.5.21.Final-redhat-1-bz-1653367.jar:7.5.21.Final-redhat-SNAPSHOT]
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [rt.jar:1.8.0_202]
            at java.util.concurrent.FutureTask.run(FutureTask.java:266) [rt.jar:1.8.0_202]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [rt.jar:1.8.0_202]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [rt.jar:1.8.0_202]
            at java.lang.Thread.run(Thread.java:748) [rt.jar:1.8.0_202]
            at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Subhashc
  • 53
  • 10
  • The error in generally means that the periodic recovery thread (http://jbossts.blogspot.com/2018/01/narayana-periodic-recovery-of-xa.html) is not capable to restore data from the object store. The object store is the storage where transaction data is saved. It could be a file system or database via JDBC. The recovery is run for transactions which crashed for some reason and thus they were not finished (with commit or rollback). The error itself is pretty short to provide some help. Could you elaborate a little bit more? – chalda Feb 10 '20 at 14:46
  • What version of WildFly/JBoss EAP do you use? When does this occur - is during application server startup or just during the normal work? Are there some prior errors in the server log before this one (e.g. hard drive is not available or connection to database is not ready)? What object store do you use (you need to check in the app server configuration, e.g. https://wildscribe.github.io/WildFly/18.0/subsystem/transactions/index.html#attr-use-jdbc-store)? How often do you experience this error? – chalda Feb 10 '20 at 14:49
  • Hi @chalda thanks for the comment, i use JBoss AS release: 7.5.21. This occur when restarted my jboss instance, during the startup. I did not notice any error in the server prior to this. I use the local file system for the object-store. This is intermittent issue, very difficult to reproduce. – Subhashc Feb 12 '20 at 17:09
  • Hi @Subhashc. Even I can't tell you with all certainty, as I can't find the particular issue in the WFLY issue tracker (https://issues.redhat.com/projects/WFLY/issues/), I think it's a trouble in ordering of subsystem startup. The JBoss/WildFly defines ordering for the subsystem when they are starting. ORB needs to be started before the transaction periodic recovery starts processing. As you use quite old version of the app server I assume the ordering could have a bug in ordering. I would recommend to switch to some of the more up-to-date WildFly server release (https://wildfly.org) – chalda Feb 13 '20 at 15:11
  • This is the line I infer my assumption from: `FATAL [com.arjuna.ats.jts] (Periodic Recovery) ARJUNA022006: The ORB has not been initialized yet E`. It says that periodic recovery tries to access the ORB but the ORB was not initialized yet. Initialization of ORB is responsibility of IIOP subsystem which has to be started before the periodic recovery starts. – chalda Feb 13 '20 at 15:13
  • This issue is intermittent, and we have observed it first time while starting. Now even we can see this issue when a remote ejb client calls the ejb method, this error is seen in the ejb server. Will update the description with the stack trace – Subhashc Feb 14 '20 at 09:23
  • ok. if I understand right you added the error stacktrace which shows that the 'com.arjuna.ats.internal.jts.ORBManager.getPOA(ORBManager.java:96)' fails with the fatal error, right? Is that all or you can provide some more details? From my perspective this new trace only justifies that there is and issue of the ORB initialization. It says that it's not capable to call the remote EJB via IIOP orb. Which is logical when the ORB was not correctly activated. Do you have some more details or have you tested it with newer WildFly version? – chalda Feb 24 '20 at 11:58
  • Hi @chalda, thank you for your reply, we have identify the cause of that error but the actual root cause is not identified. I have created a new question https://stackoverflow.com/questions/60378505/jboss-use-jts-instead-of-jta which details the issue. I will update this ticket with the answer once I get a clarification on new ticket. – Subhashc Feb 24 '20 at 14:58

0 Answers0