1

Can somebody elabarate some strange issue I ran into with Spring Boot? I have a reactive Spring MVC Controller such as:

private final Scheduler dvScheduler = Schedulers.newParallel("pool", 1000);

@Override
@SneakyTrows
@GetMapping
public Mono<String> method(Mono<String> mono) {
    return mono
        .publishOn(dvScheduler)
        .flatMap(body -> {
            Thread.sleep(8000L);
            return Mono.just("");
        });
}

This is an external system stub for testing the main system in cases of failure (8sec delay emulates external system's high reponse time). The goal is to measure main system's throughput in 'almost failing' cases such as high reponse time. So the stub is sleeping emulating high response time while the main system calls it using web-client.

The problem is that this app (the stub) can serve no more than 200 concurrent requests no matter how I configure the thread pool. E.g. using tomcat:

  tomcat:
    threads:
      max: 1000
      min-spare: 1000
    accept-count: 1000
    max-connections: 10000

I can run some profiler (or get a thread dump) and see that all 1000 threads are created but only 200 of them are staying on Thread.sleep() method from my controller. All other 800 are free in the thread pool. The load is simulated using Soap-UI load test where I can see something about 15-20tps due to this issue.

I am using Spring Boot 2.3.4.RELEASE, tested with Tomcat, Netty and Undertow both reactive and classic approach. The result is always the same - a bottleneck with 200 concurrent requests.

800 threads like this:

 jdk.internal.misc.Unsafe.park(boolean, long)
    java.util.concurrent.locks.LockSupport.park(java.lang.Object) (line: 194)
    java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await() (line: 2081)
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() (line: 1170)
    java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() (line: 899)
    java.util.concurrent.ThreadPoolExecutor.getTask() (line: 1054)
    java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) (line: 1114)
    java.util.concurrent.ThreadPoolExecutor$Worker.run() (line: 628)
    java.lang.Thread.run() (line: 834)

200 threads

java.lang.Thread.sleep(long)
java.lang.Thread.sleep(long, int) (line: 339)
my.org.controller.stub.StubController.method(Mono<String>) (line: 111)

The idea is just to make sure that the stub app will not limit TPS of the main system.

What restricts Spring Boot app's throughput to such a number? Where could be that setting = 200?

UPDATE: Seems a Soap-UI issue. After switching to JMeter TPS is higher and the threads are busy in I/O operations. I have -Dsoapui.threadpool.max=500 in its properties but for some reason the tool restricts something and 400 threads with 0 delay in Soap-UI result in 20tps while the same 400 in JMeter - 50tps.

ike3
  • 1,760
  • 1
  • 17
  • 26
  • 1
    "All other 800 are blocked by something." What are they blocked by? You said you ran a profiler / thread dump, so you should have the information. You also seem to put those numbers quite high, are you running on a high-end server to warrant values over 200? Tomcat's HTTP connector does have a default of 200 threads, but then that's not an unlikely default for others either. Double check all your things and look at the information you have. – Kayaman Feb 11 '21 at 09:36
  • Please check this link - https://stackoverflow.com/questions/39002090/spring-boot-limit-on-number-of-connections-created . May be helpful. – Md Kawser Habib Feb 11 '21 at 09:39
  • I've added the stacktraces of sleeping threads. The value 1000 is just random to make sure I am higher than 200. I am wondering why the limit is 200 because if I reduce thread pool sizes below 200, the throughput is less as well but increasing over 200 has no effect. Resources such as CPU, memory are fine too. – ike3 Feb 11 '21 at 10:05
  • Seems I don't understand why the test is flawed. Thread.sleep() is inside a stub (not in the main app). There are no changes in the main app with webclient. So the stub should respond within 8 sec while the main app is waiting for it. The problem is that due to 200 req limiting (in the stub!) the stub response is higher than 8 sec. So my goal is to make sure stub is responging with constant 8sec but with higher TPS due to higher number of concurrent requests. – ike3 Feb 11 '21 at 10:42
  • 1
    You have yourself an old fashioned configuration mystery on your hands. You could try [enabling JMX](https://stackoverflow.com/questions/61053979/how-to-enable-jmx-for-embedded-tomcat-in-spring-boot-application) and check out the runtime values. – Kayaman Feb 11 '21 at 11:34
  • Seems I was completely wrong and the issue is not in java but in testing framework. After switching from Soap-UI to JMeter I am getting different results and the threads are now busy. – ike3 Feb 11 '21 at 13:52

0 Answers0