Tomcat stop serving when spring boot application receive first paralel requests

Question

I have spring boot application that accept requests and send this requests to soap server using open feign and returns response back to the user. This application accepts Average 200 request per second. When I send first request and wait for response for a long time it warms up and serve normally after that. Problem start occurs when it start to accept many requests from the beginning. Application stops the serving. None of user could send or receive request/response.

I checked the threads using jxm and found the all of the http-nio-8080-exec are in blocked stage:

Here is the stack trace of the thread:

Name: http-nio-8080-exec-107
State: BLOCKED on sun.net.www.protocol.jar.URLJarFile@d706f19 owned by: http-nio-8080-exec-9
Total blocked: 6,797  Total waited: 0

Stack trace: 
java.util.zip.ZipFile.getEntry(ZipFile.java:314)
java.util.jar.JarFile.getEntry(JarFile.java:240)
sun.net.www.protocol.jar.URLJarFile.getEntry(URLJarFile.java:128)
sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:134)
sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:152)
java.net.URL.openStream(URL.java:1057)
java.lang.ClassLoader.getSystemResourceAsStream(ClassLoader.java:1331)
com.sun.xml.internal.bind.v2.runtime.reflect.opt.AccessorInjector.tailor(AccessorInjector.java:103)
com.sun.xml.internal.bind.v2.runtime.reflect.opt.AccessorInjector.prepare(AccessorInjector.java:68)
com.sun.xml.internal.bind.v2.runtime.reflect.opt.OptimizedAccessorFactory.get(OptimizedAccessorFactory.java:164)
com.sun.xml.internal.bind.v2.runtime.reflect.Accessor$FieldReflection.optimize(Accessor.java:271)
com.sun.xml.internal.bind.v2.runtime.reflect.TransducedAccessor$CompositeTransducedAccessorImpl.<init>(TransducedAccessor.java:220)
com.sun.xml.internal.bind.v2.runtime.reflect.TransducedAccessor.get(TransducedAccessor.java:160)
com.sun.xml.internal.bind.v2.runtime.property.AttributeProperty.<init>(AttributeProperty.java:76)
com.sun.xml.internal.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:93)
com.sun.xml.internal.bind.v2.runtime.ClassBeanInfoImpl.<init>(ClassBeanInfoImpl.java:166)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:488)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:507)
com.sun.xml.internal.bind.v2.runtime.property.SingleElementNodeProperty.<init>(SingleElementNodeProperty.java:90)
sun.reflect.GeneratedConstructorAccessor201.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
com.sun.xml.internal.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:113)
com.sun.xml.internal.bind.v2.runtime.ClassBeanInfoImpl.<init>(ClassBeanInfoImpl.java:166)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:488)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:507)
com.sun.xml.internal.bind.v2.runtime.property.SingleElementNodeProperty.<init>(SingleElementNodeProperty.java:90)
sun.reflect.GeneratedConstructorAccessor201.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
com.sun.xml.internal.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:113)
com.sun.xml.internal.bind.v2.runtime.ClassBeanInfoImpl.<init>(ClassBeanInfoImpl.java:166)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:488)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:507)
com.sun.xml.internal.bind.v2.runtime.property.SingleElementNodeProperty.<init>(SingleElementNodeProperty.java:90)
sun.reflect.GeneratedConstructorAccessor201.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
com.sun.xml.internal.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:113)
com.sun.xml.internal.bind.v2.runtime.ClassBeanInfoImpl.<init>(ClassBeanInfoImpl.java:166)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:488)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:507)
com.sun.xml.internal.bind.v2.runtime.property.ArrayElementProperty.<init>(ArrayElementProperty.java:97)
com.sun.xml.internal.bind.v2.runtime.property.ArrayElementNodeProperty.<init>(ArrayElementNodeProperty.java:47)
sun.reflect.GeneratedConstructorAccessor235.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
com.sun.xml.internal.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:113)
com.sun.xml.internal.bind.v2.runtime.ClassBeanInfoImpl.<init>(ClassBeanInfoImpl.java:166)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:488)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:507)
com.sun.xml.internal.bind.v2.runtime.property.SingleElementNodeProperty.<init>(SingleElementNodeProperty.java:90)
sun.reflect.GeneratedConstructorAccessor201.newInstance(Unknown Source)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
java.lang.reflect.Constructor.newInstance(Constructor.java:423)
com.sun.xml.internal.bind.v2.runtime.property.PropertyFactory.create(PropertyFactory.java:113)
com.sun.xml.internal.bind.v2.runtime.ClassBeanInfoImpl.<init>(ClassBeanInfoImpl.java:166)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.getOrCreate(JAXBContextImpl.java:488)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:305)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl.<init>(JAXBContextImpl.java:124)
com.sun.xml.internal.bind.v2.runtime.JAXBContextImpl$JAXBContextBuilder.build(JAXBContextImpl.java:1123)
com.sun.xml.internal.bind.v2.ContextFactory.createContext(ContextFactory.java:147)
sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:247)
javax.xml.bind.ContextFinder.newInstance(ContextFinder.java:234)
javax.xml.bind.ContextFinder.find(ContextFinder.java:462)
javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:641)
javax.xml.bind.JAXBContext.newInstance(JAXBContext.java:584)
javax.xml.bind.JAXB$Cache.<init>(JAXB.java:112)
javax.xml.bind.JAXB.getContext(JAXB.java:139)
javax.xml.bind.JAXB.unmarshal(JAXB.java:259)
az.unibank.processing.rtp.config.SOAPDecoder.decode(SOAPDecoder.java:47)
feign.AsyncResponseHandler.decode(AsyncResponseHandler.java:115)
feign.AsyncResponseHandler.handleResponse(AsyncResponseHandler.java:87)
feign.SynchronousMethodHandler.executeAndDecode(SynchronousMethodHandler.java:138)
feign.SynchronousMethodHandler.invoke(SynchronousMethodHandler.java:89)
feign.ReflectiveFeign$FeignInvocationHandler.invoke(ReflectiveFeign.java:100)
com.sun.proxy.$Proxy96.calculate(Unknown Source)
az.unibank.processing.rtp.service.impl.CardServiceImpl.getCardInfoDetails(CardServiceImpl.java:264)
az.unibank.processing.rtp.service.impl.CardServiceImpl.getRiskGroup(CardServiceImpl.java:102)
az.unibank.processing.rtp.web.rest.CardController.GetRiskGroup(CardController.java:38)
sun.reflect.GeneratedMethodAccessor143.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:197)
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:141)
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:894)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1060)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:962)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
javax.servlet.http.HttpServlet.service(HttpServlet.java:626)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
javax.servlet.http.HttpServlet.service(HttpServlet.java:733)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:93)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:374)
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888)
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597)
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
   - locked org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper@650cd2c
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
java.lang.Thread.run(Thread.java:748)

It wasn't any problem at the begining. We first sent warm up requests after deploy and then gave access public but application starts to grow bigger and bigger. Now even warm up requests is not helping us.

Using spring boot version: 2.4.2

Rest controller:

@RestController
@RequestMapping("/ws/rest")
@RequiredArgsConstructor
public class CardController {

@GetMapping(value = "/MiniCardInfo/card/{card}", produces = MediaType.APPLICATION_XML_VALUE)
    public ResponseEntity<CardInfo> getCardInfo(@PathVariable String card) {
        return ResponseEntity.ok(cardService.getCardInfo(card));
    }

...
}

Simple service class:

@Service
@RequiredArgsConstructor
public class CardServiceImpl implements CardService {

@Override
    public CardInfo getCardInfo(String cardNo) {
        String body = "OUR SOAP XML";
        Tran tran = compasFeignClient.calculate(body).getCardInfo();
  }
}

Feign client:

@FeignClient(
        name = "calculatorServer",
        url = "${application.compas-url}",
        configuration = FeignSoapClientConfig.class)
public interface CompasFeignClient {

    @PostMapping(value = "", consumes = MediaType.TEXT_XML_VALUE, produces = MediaType.TEXT_XML_VALUE)
    com.tranzaxis.schemas.tran.Tran calculate(@RequestBody String request);
}

Our bad SOAP decoder class:

@FeignClient(
        name = "calculatorServer",
        url = "${application.compas-url}",
        configuration = FeignSoapClientConfig.class)
public interface CompasFeignClient {

    @PostMapping(value = "", consumes = MediaType.TEXT_XML_VALUE, produces = MediaType.TEXT_XML_VALUE)
    com.tranzaxis.schemas.tran.Tran calculate(@RequestBody String request);
}

We are creating this message Factory in configuration class:

@Primary
@Configuration
public class FeignSoapClientConfig {

    @Bean
    @Primary
    public Decoder feignDecoder() throws SOAPException {
        final MessageFactory factory = MessageFactory.newInstance();
        return new SOAPDecoder(factory);
    }
}

I can't figure out how to solve this problem. This soap decoding code may be cos problem.