Spring boot REST service stops after some time

Question

After triggering the spring boot REST service, the service runs OK for hours with all requests working without any issues. What happens is that randomly after some time, it stops. When looking into the logs I find no errors, except information that the application has been destroyed.

Logs after some time

2016-03-09 17:07:33.488  INFO 28359 --- [Thread-12] ationConfigEmbeddedWebApplicationContext : Closing org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@5df2d54c: startup date [Wed Mar 09 10:45:18 UTC 2016]; root of context hierarchy
2016-03-09 17:07:33.490  INFO 28359 --- [Thread-12] o.s.j.e.a.AnnotationMBeanExporter        : Unregistering JMX-exposed beans on shutdown
2016-03-09 17:07:33.495  INFO 28359 --- [Thread-12] o.eclipse.jetty.server.ServerConnector   : Stopped ServerConnector@585ebc93{SSL-HTTP/1.1}{0.0.0.0:8443}
2016-03-09 17:07:33.496  INFO 28359 --- [Thread-12] application                              : Destroying Spring FrameworkServlet 'dispatcherServlet'
2016-03-09 17:07:33.496  INFO 28359 --- [Thread-12] o.e.jetty.server.handler.ContextHandler  : Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@3d3969ea{/,file:/tmp/jetty-docbase.5267015092605924805.8443/,UNAVAILABLE}

Maven dependencies

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-tomcat</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.json</groupId>
            <artifactId>json</artifactId>
        </dependency>

        <dependency>
            <groupId>org.apache.hive</groupId>
            <artifactId>hive-jdbc</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.eclipse.jetty.orbit</groupId>
                    <artifactId>javax.servlet</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.eclipse.jetty.aggregate</groupId>
                    <artifactId>jetty-all</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.google.guava</groupId>
                    <artifactId>guava</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>org.apache.drill.exec</groupId>
            <artifactId>drill-jdbc</artifactId>
            <exclusions>
                <exclusion>
                    <artifactId>hadoop-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-client</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <!-- used to update stopwatch -->
            <groupId>org.javassist</groupId>
            <artifactId>javassist</artifactId>
            <version>3.20.0-GA</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.apache.drill</groupId>
            <artifactId>drill-common</artifactId>
            <exclusions>
                <exclusion>
                    <artifactId>hadoop-common</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <artifactId>hadoop-client</artifactId>
                    <groupId>org.apache.hadoop</groupId>
                </exclusion>
                <exclusion>
                    <groupId>com.google.guava</groupId>
                    <artifactId>guava</artifactId>
                </exclusion>
            </exclusions>
        </dependency>


    </dependencies>

Any ideas to why the spring boot REST API might stop? My maven dependencies are according to the demos - and it's successfully being run - it's why the service stops after random time that doesn't make sense.

The logging settings I am using for spring boot is

logging.level.org.springframework.web = DEBUG
logging.level.org.hibernate = ERROR
logging.file = /home/REST/Logging.log

No, but if that's the case I would find hibernate errors in the logs. — Mez, Mar 16 '16 at 12:37
Increase the loglevel and check what is causing the shutdown. Since it is shutting down cleanly I suspect something like a shutdown request to the embedded servlet container or spring container. — Thomas, Mar 16 '16 at 13:10
Logging is logging.level.root=WARN logging.level.org.springframework.web=DEBUG logging.level.org.hibernate=ERROR - it should show me all debugging logs — Mez, Mar 16 '16 at 13:56
@PatrickGrimard nope it's really erratic. Settings for logging are as specified in my question. Those should have all details of errors/stopping of applications right? — Mez, Mar 16 '16 at 15:20
You would think... How are you running your Spring Boot application? — Patrick Grimard, Mar 16 '16 at 15:56
My guess FWIW is a library calling `System.exit()`. Try an app with just Spring (no hadoop etc.) to see if it suffers the same fate. — Dave Syer, Mar 17 '16 at 22:10
I've the same problem. I've 2 different webservices running on same server. I've kept both the jars in a different folder. but one of the services stops after running smoothly for 2-3 days. No error logs are there. — kb4shubham, Oct 18 '16 at 07:53

score 1 · Answer 1 · answered Jan 25 '18 at 12:55

1

Root cause was my staling connection to MySQL

spring.xxx.datasource.testOnBorrow=true
spring.xxx.datasource.validationQuery=SELECT 1

More details here https://docs.spring.io/spring-ldap/docs/1.3.2.RELEASE/reference/html/pooling.html

answered Jan 25 '18 at 12:55

Mez

4,666
4
29
57

Hey Mez, I am facing same problem, can you let me know how do you solve this. May I need to delete the lines from application.properties mentioned above. Please share your knowledge. Thanks in advance. – sib10 Dec 04 '18 at 05:26
@Mez thanks... :) I tried an opposite approach by [disabling datasource](https://stackoverflow.com/questions/36387265/disable-all-database-related-auto-configuration-in-spring-boot) just for testing. Its working fine now. – Sahil Gulati Oct 29 '20 at 05:32

Spring boot REST service stops after some time

1 Answers1