High CPU utilization in Spring Cloud Loadbalancer application

Question

after the redeploy operation of my application the cpu rises to 100%. So I have to restart Tomcat to free the cpu.

The the thread that consumes all the cpu is:


    "reactor-http-epoll-1" #1569412 daemon prio=5 os_prio=0 tid=0x00007f53c010b000 nid=0x6155 runnable [0x00007f539237d000]
       java.lang.Thread.State: RUNNABLE
            at java.lang.Throwable.getStackTraceElement(Native Method)
            at java.lang.Throwable.getOurStackTrace(Throwable.java:827)
            - locked  (a java.lang.NoClassDefFoundError)
            at java.lang.Throwable.getStackTrace(Throwable.java:816)
            at ch.qos.logback.classic.spi.ThrowableProxy.(ThrowableProxy.java:55)
            at ch.qos.logback.classic.spi.LoggingEvent.(LoggingEvent.java:119)
            at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:419)
            at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
            at ch.qos.logback.classic.Logger.log(Logger.java:765)
            at io.netty.util.internal.logging.LocationAwareSlf4JLogger.log(LocationAwareSlf4JLogger.java:50)
            at io.netty.util.internal.logging.LocationAwareSlf4JLogger.warn(LocationAwareSlf4JLogger.java:202)
            at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
            at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
            at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:387)
            at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
            at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
            at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
            at java.lang.Thread.run(Thread.java:748)

The problem occurs only in redeploy operation. I replace - from shell - the war with the new version.

OS: RedHat 7.6 x86_64 (3.10.0-957.5.1.el7.x86_64)
Processor: 1 x Intel(R) Xeon(R) CPU E5-2687W v4 @ 3.00GHz
OpenJDK Runtime Environment (build 1.8.0_191-b12)
OpenJDK 64-Bit Server VM (build 25.191-b12, mixed mode)
Server version: Apache Tomcat/9.0.7.redhat-12
Server built: Sep 7 2018 13:37:03 UTC
Server number: 9.0.7.0
OS Name: Linux
OS Version: 3.10.0-957.5.1.el7.x86_64
Architecture: amd64
JVM Version: 1.8.0_191-b12
JVM Vendor: Oracle Corporation

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-dependencies</artifactId>
        <version>Hoxton.SR9</version>
        <type>pom</type>
        <scope>import</scope>
    </dependency>

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-loadbalancer</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-commons</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-webflux</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-security</artifactId>
    </dependency>

    <dependency>
        <groupId>org.thymeleaf.extras</groupId>
        <artifactId>thymeleaf-extras-springsecurity5</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-thymeleaf</artifactId>
    </dependency>

    <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-openfeign</artifactId>
        <exclusions>
            <exclusion>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-netflix-ribbon</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-tomcat</artifactId>
        <scope>provided</scope>
    </dependency>

The loadbalancer config using webflux (webclient).


    @Configuration
    @EnableAutoConfiguration
    @LoadBalancerClients({ @LoadBalancerClient(value = "${loadbalancer.client.name}") })
    public class LoadBalancerConfig implements WebMvcConfigurer {
    
        @LoadBalanced
        @Bean
        RestTemplate restClient() {
            return new RestTemplate();
        }
    
        @Bean
        public ServiceInstanceListSupplier discoveryClientServiceInstanceListSupplier(DiscoveryClient discoveryClient, Environment env, ApplicationContext context, LoadBalancerProperties loadBalancerProperties) {
            DiscoveryClientServiceInstanceListSupplier firstDelegate = new DiscoveryClientServiceInstanceListSupplier(discoveryClient, env);
    
            HealthCheckServiceInstanceListSupplier delegate = new HealthCheckServiceInstanceListSupplier(firstDelegate, loadBalancerProperties.getHealthCheck(), WebClient.create());
    
            /*
             * ObjectProvider<LoadBalancerCacheManager> cacheManagerProvider = context
             * .getBeanProvider(LoadBalancerCacheManager.class);
             *
             * if (cacheManagerProvider.getIfAvailable() != null) { return new
             * CachingServiceInstanceListSupplier(delegate,
             * cacheManagerProvider.getIfAvailable()); }
             */
            return delegate;
        }
    
    }

dependency:tree


    [INFO] 
    [INFO] -----------------------------------------------
    [INFO] Building webapp 1.0.1                                              
    [INFO] --------------------------------[ war ]---------------------------------
    [INFO] 
    [INFO] --- maven-dependency-plugin:3.1.2:tree (default-cli) @ webapp ---
    [INFO] it.xxx:webapp:war:1.0.1
    [INFO] +- org.springframework.boot:spring-boot-starter-actuator:jar:2.3.0.RELEASE:compile
    [INFO] |  +- org.springframework.boot:spring-boot-starter:jar:2.3.0.RELEASE:compile
    [INFO] |  |  +- org.springframework.boot:spring-boot:jar:2.3.0.RELEASE:compile
    [INFO] |  |  +- org.springframework.boot:spring-boot-autoconfigure:jar:2.3.0.RELEASE:compile
    [INFO] |  |  +- org.springframework.boot:spring-boot-starter-logging:jar:2.3.0.RELEASE:compile
    [INFO] |  |  |  +- ch.qos.logback:logback-classic:jar:1.2.3:compile
    [INFO] |  |  |  |  \- ch.qos.logback:logback-core:jar:1.2.3:compile
    [INFO] |  |  |  +- org.apache.logging.log4j:log4j-to-slf4j:jar:2.13.2:compile
    [INFO] |  |  |  |  \- org.apache.logging.log4j:log4j-api:jar:2.13.2:compile
    [INFO] |  |  |  \- org.slf4j:jul-to-slf4j:jar:1.7.30:compile
    [INFO] |  |  \- org.yaml:snakeyaml:jar:1.26:compile
    [INFO] |  +- org.springframework.boot:spring-boot-actuator-autoconfigure:jar:2.3.0.RELEASE:compile
    [INFO] |  |  +- org.springframework.boot:spring-boot-actuator:jar:2.3.0.RELEASE:compile
    [INFO] |  |  +- com.fasterxml.jackson.core:jackson-databind:jar:2.11.0:compile
    [INFO] |  |  |  +- com.fasterxml.jackson.core:jackson-annotations:jar:2.11.0:compile
    [INFO] |  |  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.11.0:compile
    [INFO] |  |  \- com.fasterxml.jackson.datatype:jackson-datatype-jsr310:jar:2.11.0:compile
    [INFO] |  \- io.micrometer:micrometer-core:jar:1.5.1:compile
    [INFO] |     +- org.hdrhistogram:HdrHistogram:jar:2.1.12:compile
    [INFO] |     \- org.latencyutils:LatencyUtils:jar:2.0.3:runtime
    [INFO] +- org.springframework.boot:spring-boot-starter-web:jar:2.3.0.RELEASE:compile
    [INFO] |  +- org.springframework.boot:spring-boot-starter-json:jar:2.3.0.RELEASE:compile
    [INFO] |  |  +- com.fasterxml.jackson.datatype:jackson-datatype-jdk8:jar:2.11.0:compile
    [INFO] |  |  \- com.fasterxml.jackson.module:jackson-module-parameter-names:jar:2.11.0:compile
    [INFO] |  +- org.springframework:spring-web:jar:5.2.6.RELEASE:compile
    [INFO] |  |  \- org.springframework:spring-beans:jar:5.2.6.RELEASE:compile
    [INFO] |  \- org.springframework:spring-webmvc:jar:5.2.6.RELEASE:compile
    [INFO] |     \- org.springframework:spring-expression:jar:5.2.6.RELEASE:compile
    [INFO] +- org.springframework.boot:spring-boot-starter-security:jar:2.3.0.RELEASE:compile
    [INFO] |  +- org.springframework:spring-aop:jar:5.2.6.RELEASE:compile
    [INFO] |  +- org.springframework.security:spring-security-config:jar:5.3.2.RELEASE:compile
    [INFO] |  |  \- org.springframework.security:spring-security-core:jar:5.3.2.RELEASE:compile
    [INFO] |  \- org.springframework.security:spring-security-web:jar:5.3.2.RELEASE:compile
    [INFO] +- org.thymeleaf.extras:thymeleaf-extras-springsecurity5:jar:3.0.4.RELEASE:compile
    [INFO] |  \- org.slf4j:slf4j-api:jar:1.7.30:compile
    [INFO] +- org.springframework.boot:spring-boot-starter-thymeleaf:jar:2.3.0.RELEASE:compile
    [INFO] |  +- org.thymeleaf:thymeleaf-spring5:jar:3.0.11.RELEASE:compile
    [INFO] |  |  \- org.thymeleaf:thymeleaf:jar:3.0.11.RELEASE:compile
    [INFO] |  |     +- org.attoparser:attoparser:jar:2.0.5.RELEASE:compile
    [INFO] |  |     \- org.unbescape:unbescape:jar:1.1.6.RELEASE:compile
    [INFO] |  \- org.thymeleaf.extras:thymeleaf-extras-java8time:jar:3.0.4.RELEASE:compile
    [INFO] +- org.springframework.boot:spring-boot-starter-test:jar:2.3.0.RELEASE:test
    [INFO] |  +- org.springframework.boot:spring-boot-test:jar:2.3.0.RELEASE:test
    [INFO] |  +- org.springframework.boot:spring-boot-test-autoconfigure:jar:2.3.0.RELEASE:test
    [INFO] |  +- com.jayway.jsonpath:json-path:jar:2.4.0:test
    [INFO] |  |  \- net.minidev:json-smart:jar:2.3:test
    [INFO] |  |     \- net.minidev:accessors-smart:jar:1.2:test
    [INFO] |  |        \- org.ow2.asm:asm:jar:5.0.4:test
    [INFO] |  +- jakarta.xml.bind:jakarta.xml.bind-api:jar:2.3.3:test
    [INFO] |  |  \- jakarta.activation:jakarta.activation-api:jar:1.2.2:test
    [INFO] |  +- org.assertj:assertj-core:jar:3.16.1:test
    [INFO] |  +- org.hamcrest:hamcrest:jar:2.2:test
    [INFO] |  +- org.junit.jupiter:junit-jupiter:jar:5.6.2:test
    [INFO] |  |  +- org.junit.jupiter:junit-jupiter-api:jar:5.6.2:test
    [INFO] |  |  |  +- org.opentest4j:opentest4j:jar:1.2.0:test
    [INFO] |  |  |  \- org.junit.platform:junit-platform-commons:jar:1.6.2:test
    [INFO] |  |  +- org.junit.jupiter:junit-jupiter-params:jar:5.6.2:test
    [INFO] |  |  \- org.junit.jupiter:junit-jupiter-engine:jar:5.6.2:test
    [INFO] |  +- org.junit.vintage:junit-vintage-engine:jar:5.6.2:test
    [INFO] |  |  +- org.apiguardian:apiguardian-api:jar:1.1.0:test
    [INFO] |  |  +- org.junit.platform:junit-platform-engine:jar:1.6.2:test
    [INFO] |  |  \- junit:junit:jar:4.13:test
    [INFO] |  +- org.mockito:mockito-core:jar:3.3.3:test
    [INFO] |  |  +- net.bytebuddy:byte-buddy:jar:1.10.10:test
    [INFO] |  |  +- net.bytebuddy:byte-buddy-agent:jar:1.10.10:test
    [INFO] |  |  \- org.objenesis:objenesis:jar:2.6:test
    [INFO] |  +- org.mockito:mockito-junit-jupiter:jar:3.3.3:test
    [INFO] |  +- org.skyscreamer:jsonassert:jar:1.5.0:test
    [INFO] |  |  \- com.vaadin.external.google:android-json:jar:0.0.20131108.vaadin1:test
    [INFO] |  +- org.springframework:spring-core:jar:5.2.6.RELEASE:compile
    [INFO] |  |  \- org.springframework:spring-jcl:jar:5.2.6.RELEASE:compile
    [INFO] |  +- org.springframework:spring-test:jar:5.2.6.RELEASE:test
    [INFO] |  \- org.xmlunit:xmlunit-core:jar:2.7.0:test
    [INFO] +- org.webjars:bootstrap:jar:4.2.1:compile
    [INFO] |  +- org.webjars:jquery:jar:3.0.0:compile
    [INFO] |  \- org.webjars:popper.js:jar:1.14.3:compile
    [INFO] +- org.springframework.cloud:spring-cloud-starter-openfeign:jar:2.2.6.RELEASE:compile
    [INFO] |  +- org.springframework.cloud:spring-cloud-starter:jar:2.2.6.RELEASE:compile
    [INFO] |  |  +- org.springframework.cloud:spring-cloud-context:jar:2.2.6.RELEASE:compile
    [INFO] |  |  \- org.springframework.security:spring-security-rsa:jar:1.0.9.RELEASE:compile
    [INFO] |  |     \- org.bouncycastle:bcpkix-jdk15on:jar:1.64:compile
    [INFO] |  |        \- org.bouncycastle:bcprov-jdk15on:jar:1.64:compile
    [INFO] |  +- org.springframework.cloud:spring-cloud-openfeign-core:jar:2.2.6.RELEASE:compile
    [INFO] |  |  +- org.springframework.boot:spring-boot-starter-aop:jar:2.3.0.RELEASE:compile
    [INFO] |  |  |  \- org.aspectj:aspectjweaver:jar:1.9.5:compile
    [INFO] |  |  \- io.github.openfeign.form:feign-form-spring:jar:3.8.0:compile
    [INFO] |  |     +- io.github.openfeign.form:feign-form:jar:3.8.0:compile
    [INFO] |  |     \- commons-fileupload:commons-fileupload:jar:1.4:compile
    [INFO] |  +- io.github.openfeign:feign-core:jar:10.10.1:compile
    [INFO] |  +- io.github.openfeign:feign-slf4j:jar:10.10.1:compile
    [INFO] |  \- io.github.openfeign:feign-hystrix:jar:10.10.1:compile
    [INFO] |     +- com.netflix.archaius:archaius-core:jar:0.7.6:compile
    [INFO] |     |  +- com.google.code.findbugs:jsr305:jar:3.0.1:runtime
    [INFO] |     |  +- commons-configuration:commons-configuration:jar:1.8:runtime
    [INFO] |     |  |  \- commons-lang:commons-lang:jar:2.6:runtime
    [INFO] |     |  \- com.google.guava:guava:jar:29.0-jre:runtime
    [INFO] |     |     +- com.google.guava:failureaccess:jar:1.0.1:runtime
    [INFO] |     |     +- com.google.guava:listenablefuture:jar:9999.0-empty-to-avoid-conflict-with-guava:runtime
    [INFO] |     |     +- org.checkerframework:checker-qual:jar:2.11.1:runtime
    [INFO] |     |     +- com.google.errorprone:error_prone_annotations:jar:2.3.4:runtime
    [INFO] |     |     \- com.google.j2objc:j2objc-annotations:jar:1.3:runtime
    [INFO] |     \- com.netflix.hystrix:hystrix-core:jar:1.5.18:compile
    [INFO] |        \- io.reactivex:rxjava:jar:1.3.8:compile
    [INFO] +- org.springframework.cloud:spring-cloud-starter-loadbalancer:jar:2.2.6.RELEASE:compile
    [INFO] |  +- org.springframework.cloud:spring-cloud-loadbalancer:jar:2.2.6.RELEASE:compile
    [INFO] |  |  +- org.springframework.boot:spring-boot-starter-validation:jar:2.3.0.RELEASE:compile
    [INFO] |  |  |  \- org.hibernate.validator:hibernate-validator:jar:6.1.5.Final:compile
    [INFO] |  |  |     +- jakarta.validation:jakarta.validation-api:jar:2.0.2:compile
    [INFO] |  |  |     +- org.jboss.logging:jboss-logging:jar:3.4.1.Final:compile
    [INFO] |  |  |     \- com.fasterxml:classmate:jar:1.5.1:compile
    [INFO] |  |  +- io.projectreactor:reactor-core:jar:3.3.5.RELEASE:compile
    [INFO] |  |  |  \- org.reactivestreams:reactive-streams:jar:1.0.3:compile
    [INFO] |  |  \- io.projectreactor.addons:reactor-extra:jar:3.3.3.RELEASE:compile
    [INFO] |  +- org.springframework.boot:spring-boot-starter-cache:jar:2.3.0.RELEASE:compile
    [INFO] |  |  \- org.springframework:spring-context-support:jar:5.2.6.RELEASE:compile
    [INFO] |  \- com.stoyanr:evictor:jar:1.0.0:compile
    [INFO] +- org.springframework.cloud:spring-cloud-commons:jar:2.2.6.RELEASE:compile
    [INFO] |  \- org.springframework.security:spring-security-crypto:jar:5.3.2.RELEASE:compile
    [INFO] +- org.springframework.boot:spring-boot-starter-webflux:jar:2.3.0.RELEASE:compile
    [INFO] |  +- org.springframework.boot:spring-boot-starter-reactor-netty:jar:2.3.0.RELEASE:compile
    [INFO] |  |  \- io.projectreactor.netty:reactor-netty:jar:0.9.7.RELEASE:compile
    [INFO] |  |     +- io.netty:netty-codec-http:jar:4.1.49.Final:compile
    [INFO] |  |     |  +- io.netty:netty-common:jar:4.1.49.Final:compile
    [INFO] |  |     |  +- io.netty:netty-buffer:jar:4.1.49.Final:compile
    [INFO] |  |     |  +- io.netty:netty-transport:jar:4.1.49.Final:compile
    [INFO] |  |     |  \- io.netty:netty-codec:jar:4.1.49.Final:compile
    [INFO] |  |     +- io.netty:netty-codec-http2:jar:4.1.49.Final:compile
    [INFO] |  |     +- io.netty:netty-handler:jar:4.1.49.Final:compile
    [INFO] |  |     |  \- io.netty:netty-resolver:jar:4.1.49.Final:compile
    [INFO] |  |     +- io.netty:netty-handler-proxy:jar:4.1.49.Final:compile
    [INFO] |  |     |  \- io.netty:netty-codec-socks:jar:4.1.49.Final:compile
    [INFO] |  |     \- io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.49.Final:compile
    [INFO] |  |        \- io.netty:netty-transport-native-unix-common:jar:4.1.49.Final:compile
    [INFO] |  +- org.springframework:spring-webflux:jar:5.2.6.RELEASE:compile
    [INFO] |  \- org.synchronoss.cloud:nio-multipart-parser:jar:1.1.0:compile
    [INFO] |     \- org.synchronoss.cloud:nio-stream-storage:jar:1.1.3:compile
    [INFO] +- it.xxx:type:jar:1.0-SNAPSHOT:compile
    [INFO] |  \- org.projectlombok:lombok:jar:1.18.12:compile
    [INFO] +- it.xxx:utility:jar:1.0-SNAPSHOT:compile
    [INFO] |  +- org.springframework:spring-context:jar:5.2.6.RELEASE:compile
    [INFO] |  +- javax.mail:mail:jar:1.4.7:compile
    [INFO] |  |  \- javax.activation:activation:jar:1.1:compile
    [INFO] |  +- org.apache.commons:commons-lang3:jar:3.10:compile
    [INFO] |  +- org.modelmapper:modelmapper:jar:2.1.0:compile
    [INFO] |  +- commons-io:commons-io:jar:2.6:compile
    [INFO] |  \- io.jsonwebtoken:jjwt:jar:0.9.1:compile
    [INFO] +- org.json:json:jar:20190722:compile
    [INFO] +- commons-validator:commons-validator:jar:1.6:compile
    [INFO] |  +- commons-beanutils:commons-beanutils:jar:1.9.2:compile
    [INFO] |  +- commons-digester:commons-digester:jar:1.8.1:compile
    [INFO] |  +- commons-logging:commons-logging:jar:1.2:compile
    [INFO] |  \- commons-collections:commons-collections:jar:3.2.2:compile
    [INFO] +- com.google.code.gson:gson:jar:2.8.5:compile
    [INFO] \- org.springframework.boot:spring-boot-starter-tomcat:jar:2.3.0.RELEASE:provided
    [INFO]    +- jakarta.annotation:jakarta.annotation-api:jar:1.3.5:compile
    [INFO]    +- org.apache.tomcat.embed:tomcat-embed-core:jar:9.0.35:provided
    [INFO]    +- org.glassfish:jakarta.el:jar:3.0.3:compile
    [INFO]    \- org.apache.tomcat.embed:tomcat-embed-websocket:jar:9.0.35:provided

Thanks!

Can you verify against Hoxton.SR9? Also, nothing in this stack points directly to the LoadBalancer. Do you have any more details that would allow for verifying the issue? — OlgaMaciaszek, Jan 13 '21 at 12:16
I added the loadbalancer configuration. The webclient (webflux) is .I will try with Hoxton.SR9. — Nietzsche, Jan 13 '21 at 14:42
The dependency that generates the cpu consumption is Netty. As the webclient (webflux) is the only one to use reactor and Netty. — Nietzsche, Jan 13 '21 at 14:43
I updated the pom and added tree dependency. I am trying with Hoxton.SR9 and it seems that the problem has been solved. However I need to do further tests to be sure. Thanks Olga. — Nietzsche, Jan 15 '21 at 10:28
Hi Olga, it seems that the problem has been resolved with the update of dependencies. Thanks! — Nietzsche, Jan 26 '21 at 08:51

High CPU utilization in Spring Cloud Loadbalancer application

0 Answers0