3

I'm running a Spring Batch app that inserts a few thousand rows in my Postgresql database per batch. Every now & then, for no apparent reason, we get the following exception:

org.springframework.orm.jpa.JpaSystemException: nested exception is javax.persistence.PersistenceException at ...

There is nothing to tell us what the PersistenceException is. It doesn't always happen, and it does not occur in the same place. I've been researching common causes of this exception and have tried everything:

  1. I implemented a JPAVendorAdapter which is supposed to give better error messages (it doesn't)
  2. I verified that the repository class was defined properly
  3. I've run it locally and put a breakpoint in the JPASystemException, but when the execution got there, there was nothing in the stacktrace to point to what went wrong.

Does anyone have any other ideas? I don't even know what code to show here, because I don't know where the error stems from, be it a bad configuration, service, or repository. I would really appreciate any more ideas!

The full stacktrace is below:

com.company.common.exception.ERDException: org.springframework.orm.jpa.JpaSystemException: 
nested exception is javax.persistence.PersistenceException 
at com.company.ingest.delegate.BaseDelegate.insertRecords(BaseDelegate.java:537) 
at com.company.ingest..delegate.Delegate.writeResult(Delegate.java:164) 
at com.company.ingest.job.AbstractBaseJob.insertReturnsRecords(AbstractBaseJob.java:678) 
at com.company.ingest.job.PerfReturnsWriter.processData(PerfReturnsWriter.java:115) 
at com.company.ingest.job.PerfReturnsWriter.write(PerfReturnsWriter.java:83) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:498) 
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) 
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) 
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) 
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133) 
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121) 
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) 
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) 
at com.sun.proxy.$Proxy201.write(Unknown Source) 
at org.springframework.batch.core.step.item.SimpleChunkProcessor.writeItems(SimpleChunkProcessor.java:175) 
at org.springframework.batch.core.step.item.SimpleChunkProcessor.doWrite(SimpleChunkProcessor.java:151) 
at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor$3.doWithRetry(FaultTolerantChunkProcessor.java:328) 
at org.springframework.retry.support.RetryTemplate.doExecute(RetryTemplate.java:287) 
at org.springframework.retry.support.RetryTemplate.execute(RetryTemplate.java:211) 
at org.springframework.batch.core.step.item.BatchRetryTemplate.execute(BatchRetryTemplate.java:217) 
at org.springframework.batch.core.step.item.FaultTolerantChunkProcessor.write(FaultTolerantChunkProcessor.java:420) 
at org.springframework.batch.core.step.item.SimpleChunkProcessor.process(SimpleChunkProcessor.java:199) 
at org.springframework.batch.core.step.item.ChunkOrientedTasklet.execute(ChunkOrientedTasklet.java:75) 
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:406) 
at org.springframework.batch.core.step.tasklet.TaskletStep$ChunkTransactionCallback.doInTransaction(TaskletStep.java:330) 
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133) 
at org.springframework.batch.core.step.tasklet.TaskletStep$2.doInChunkContext(TaskletStep.java:272) 
at org.springframework.batch.core.scope.context.StepContextRepeatCallback.doInIteration(StepContextRepeatCallback.java:81) 
at org.springframework.batch.repeat.support.RepeatTemplate.getNextResult(RepeatTemplate.java:374) 
at org.springframework.batch.repeat.support.RepeatTemplate.executeInternal(RepeatTemplate.java:215) 
at org.springframework.batch.repeat.support.RepeatTemplate.iterate(RepeatTemplate.java:144) 
at org.springframework.batch.core.step.tasklet.TaskletStep.doExecute(TaskletStep.java:257) 
at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:200) 
at org.springframework.batch.core.job.SimpleStepHandler.handleStep(SimpleStepHandler.java:148) 
at org.springframework.batch.core.job.flow.JobFlowExecutor.executeStep(JobFlowExecutor.java:64) 
at org.springframework.batch.core.job.flow.support.state.StepState.handle(StepState.java:67) 
at org.springframework.batch.core.job.flow.support.SimpleFlow.resume(SimpleFlow.java:169) 
at org.springframework.batch.core.job.flow.support.SimpleFlow.start(SimpleFlow.java:144) 
at org.springframework.batch.core.job.flow.FlowJob.doExecute(FlowJob.java:134) 
at org.springframework.batch.core.job.AbstractJob.execute(AbstractJob.java:306) 
at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:135) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
at java.lang.Thread.run(Thread.java:748)
CNDyson
  • 1,687
  • 7
  • 28
  • 63
  • Does the `PersistenceException` really have no inner exception? Usually in jpa there are like 2 - 5 more nested execptions under the PersistenceException. Does the persistence exception really have no message or are you just logging it wrong? Can you show the full stacktrace of the exception? – x4rf41 Jul 11 '20 at 12:27
  • It really has NO other exception. I don’t have the stacktrace with me right now, but I can promise you nothing is there. – CNDyson Jul 11 '20 at 12:29
  • The full stacktrace would help a lot anyway. because it would show in which class and at which line it occurred. Otherwise this is really hopeless – x4rf41 Jul 11 '20 at 12:34
  • Ok, when I get to my laptop I’ll post it. But I know the line throwing the exception is the repository.save() method. And all the “caused by” line says is “caused by PersistenceException”. – CNDyson Jul 11 '20 at 13:09
  • @x4rf41 I've added the stacktrace. Please let know if anything pops out. – CNDyson Jul 13 '20 at 14:45
  • The deepest part of the stacktrace is `at com.company.ingest.delegate.BaseDelegate.insertRecords(BaseDelegate.java:537)`. That is your own code right?. So the Exception is actually thrown from your code. What do you do there? Do you maybe ignore a thrown exception there (or not rethrowing it as an inner exception)? Like: `try { ... } catch(Exception ex) { throw MyException(/* without passing the original as inner exeception */) }` or something like that? – x4rf41 Jul 13 '20 at 23:02
  • Actually the stacktrace shows `nested exception is javax.persistence.PersistenceException`? where is the stacktrace for the nested exception? I wanted the FULL stacktrace, that means ALL nested exceptions – x4rf41 Jul 13 '20 at 23:03
  • @x4rf41 That's the problem - that's all there is!! That's all we get. Yes, we throw the exception up a few methods. The exception occurs at the repository.save() method. I'll put the try-catch around that method and see if I get anything more. – CNDyson Jul 14 '20 at 13:51
  • Your problem is obviously that you either do not log the exception correctly or you ignore the actual exception somehow. I bet the actual exception has a proper reason. The deepest point of the stack trace you posted is `at com.company.ingest.delegate.BaseDelegate.insertRecords(BaseDelegate.java:537)` which is YOUR code. If your code does not provide the full stacktrace, then fix that. You also said that the exception occurs at repository.save(), but that is not part of the stacktrace you posted! Do you have a stacktrace that includes repository.save()? – x4rf41 Jul 15 '20 at 02:41
  • Please post the contents of `com.company.ingest.delegate.BaseDelegate.insertRecords(BaseDelegate.java:537)`. It appears that the exception handling here in this method is doing something that causes you to lose information. Does it by any chance have a catch block `} catch (Exception exception) { throw new com.company.common.exception.ERDException(exception.toString());}` – Nathan Jul 17 '20 at 10:11
  • Please enable debug logs in spring it will tell transaction info too. In the application, properties add: logging.level.org.springframework.orm.jpa=DEBUG logging.level.org.springframework.transaction=DEBUG more info with an exception can only help to find out why. – Amit Vyas Jul 20 '20 at 13:27

2 Answers2

1

The problem is that you are handling the exception incorrectly in a way that suppresses the root cause (or that you declared your exception class to suppress any exception-stacktraces that were passed in). The actual exception that is thrown is your own business exception.

You didn't post the content of your method: com.company.ingest.delegate.BaseDelegate.insertRecords(BaseDelegate.java:537) but based on what you posted, I am going to guess that the method looks something like this:

public void insertRecords(...) {
    try {
        repository.save(...);
    } catch (Exception exception) {
        throw new com.company.common.exception.ERDException(exception.toString());
    }
}

This would explain why the exception that is thrown and logged does not contain any meaningful information. The exception that you caught does contain a message itself. It wraps another exception that (we assume) contains the root cause.

If my assumption about your method is correct, then the following change would provide the root cause the next time the error occurs, assuming your exception passes the parameters along to the superclass (all the way to Throwable):

public void insertRecords(...) {
    try {
        repository.save(...);
    } catch (Exception exception) {
        throw new com.company.common.exception.ERDException("Unable to insert record", exception);
    }
}
Nathan
  • 1,576
  • 8
  • 18
  • Yes, your assumption is correct! I'll will wrap the save method and see what I get. I'll accept your answer in a bit, thanks a lot. – CNDyson Jul 17 '20 at 14:22
  • It looks like you were right! Thanks for your help. – CNDyson Jul 20 '20 at 19:20
  • For what it's worth, we're still seeing this exception. I thought we'd gotten rid of it, but it's still cropping up. Do you have any other ideas? – CNDyson Aug 07 '20 at 15:04
  • Did you make the change that I proposed? If so, then the new stack trace provides more information. Nobody can help without the information my recommended change would provide. – Nathan Aug 17 '20 at 06:37
  • Absolutely I made the change everywhere. There is no additional information in the stacktrace. – CNDyson Aug 17 '20 at 17:36
  • would need to see the current version of the code, the current stack trace, and the content of your exception class ERDException – Nathan Aug 20 '20 at 20:43
0

Use the exception methods documented here to troubleshoot the root cause for the exception in the stack trace of yours.

https://docs.oracle.com/cd/E17802_01/products/products/persistence/javadoc-1_0-fr/javax/persistence/PersistenceException.html

Ahmed Maher
  • 1,521
  • 17
  • 22
  • I'm sorry, I don't understand what you're expecting. The ExceptionTranslator class determined that the exception I'm getting isn't because of any of the known exception subclasses. – CNDyson Jul 14 '20 at 20:38
  • I mean you should do some low-level troubleshooting based on the exception thrown , check this question https://stackoverflow.com/questions/10493053/how-do-you-handle-and-parse-jpa-persistence-exceptions-to-provide-meaningfull-me – Ahmed Maher Jul 15 '20 at 07:06
  • I mean if you won't do some brute force oriented troubleshooting regarding what appears to be most likely a database related issue which raises such exception, and you don't provide the code, Then you have no other option but doing some low-level troubleshooting based on the exception thrown. – Ahmed Maher Jul 15 '20 at 07:28