2

Recently, I have upgraded the spring boot version from 2.3.4 to 2.4.2. I had below code to gracefully shutdown the application.

taskScheduler.schedule(() -> SpringApplication.exit(applicationContext, () -> 0),
            Instant.now().plus(30, ChronoUnit.SECONDS));

This piece of code was working fine before the upgrade. Now it gives the below exception and application keep running (no termination).

2021-02-08 18:56:29.530 WARN [xyz-naming-service,,] 9040 --- [ scheduling-1] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Interrupted during closing
java.lang.InterruptedException: null
at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2109)
at java.base/java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1454)
at com.zaxxer.hikari.pool.HikariPool.shutdown(HikariPool.java:255)
at com.zaxxer.hikari.HikariDataSource.close(HikariDataSource.java:351)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:339)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:273)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1152)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1145)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1111)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1080)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:171)
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1026)
at org.springframework.boot.SpringApplication.close(SpringApplication.java:1369)
at org.springframework.boot.SpringApplication.exit(SpringApplication.java:1356)
at com.xyz.renaming.Migrator.lambda$migrate$1(Migrator.java:36)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

Any idea how to resolve this issue?

Update:

Based on the @vvs answer, I change the code to close Hikari data source first. Now InterruptedException is not there. Surprisingly, the application still running. I could see below log keep logging frequently. Meaning, these thread didn't terminate.

2021-02-09 15:46:40.460 INFO [xyz-naming-service,,] 3212 --- [ool-12-thread-1] o.a.k.c.c.internals.ConsumerCoordinator : ...
Ruchira Gayan Ranaweera
  • 34,993
  • 17
  • 75
  • 115

1 Answers1

0

Think this is an issue in your application termination logic. Please have a look at this issue on HikariCP https://github.com/brettwooldridge/HikariCP/issues/1295

The resolution says

Looks like calling hikariDataSource.close() before SpringApplication.exit() fixes the issue

vvs
  • 1,066
  • 8
  • 16
  • When I use `hikariDataSource.close()` before `SpringApplication.exit()`, previous `IntteruptedException` is not there. But application still keep running. I could see below log line keep logging `2021-02-09 15:46:40.460 INFO [xyz-naming-service,,] 3212 --- [ool-12-thread-1] o.a.k.c.c.internals.ConsumerCoordinator : ...` – Ruchira Gayan Ranaweera Feb 09 '21 at 07:48
  • Do you have any other long running logic. Also please consider this question https://stackoverflow.com/questions/48572612/difference-between-exitcodegenerator-and-system-exit0 – vvs Feb 09 '21 at 16:14
  • @Ruchira Gayan Ranaweera Do you have kafka anywhere in the mix. A quick search on `o.a.k.c.c.internals.ConsumerCoordinator` suggests the log originates from kafka. – vvs Feb 10 '21 at 17:24
  • Yes. Kafka is there. – Ruchira Gayan Ranaweera Feb 11 '21 at 01:19
  • In that case I suspect you need to close it the kafka stuff same way as the hikaricp. Likely that bit is set up to survive beyond the spring boot stuff and hence doesn't close with the exit method – vvs Feb 11 '21 at 01:25