Tomcat 8 with CompletableFutures in Java 8

Question

I want to parallelise my application. I am using Tomcat8 to deploy my web application. I am using Tomcat Default settings(HTTP Connector Thread count 200 and Default JVM settings). I want to use CompletableFuture in Java to complete the task in parallel. Ex - If I have 3 tasks TASK1,TASK2,TASK3 then instead of executing them in sequence I want to execute each task in separate threads using CompletableFuture and combine results. My question is, at any point of time, Tomcat receives 200 request, How many Threads are safe to create in Executor? If Executors.newFixedThreadPool(600), is 600 is a good number because, at any point of time I get 200 request and three parallel task to complete so I need minimum 600 threads(Theoretically). I feel creating more Threads may loose performance.

Answer 1

How many threads you can create depends on many factors, mainly in the specifications of the machine and OS.

This answer says.

This depends on the CPU you're using, on the OS, on what other processes are doing, on what Java release you're using, and other factors. I've seen a Windows server have > 6500 Threads before bringing the machine down.

I personally have used almost 1000 threads, and the performance of my machine was still good.

About using Executors.newFixedThreadPool(600) you have to analize if that is the better executor type that fits your application caracteristics and needs.

Here you can see a comparison between FixedThreadPool and CachedThreadPool:

FixedThreadPool vs CachedThreadPool: the lesser of two evils

If the constant thread pool (of 600), will have most of the threads idle most of the time, you can use the chache thread pool that will create as many threads as necessary and then will keep them for certain time or as long as they continue to being used. You would probably get a benefit of using fixed thread pool, if you have the 200 executing the 3 tasks constantly.

You can also use the CachedThreadPool with a maximun number of threads to be createad using a custom thread factory.

By the other hand, if most of the tasks are short tasks, you can use Executors.newWorkStealingPool() this ensures that your available cpu cores are always working, setting the level of parallelistm with Runtime.getRuntime().availableProcessors(), if some thread finishes its work, it can steal tasks from another threads queue.

You can see more about ForkJoinPool and Executors.newWorkStealingPool() (note: newWorkStealingPool uses ForkJoinPool internally):

Detailed difference between Java8 ForkJoinPool and Executors.newWorkStealingPool?

Answer 2

The Answer by Jose Da Silva is correct and smart. A bit more explanation here.

There is no hard-and-fast rule to follow. As others said, it depends on many factors such as the nature of the particular tasks, their duration, how CPU-intensive the tasks versus how often the tasks may be waiting on resources, how the thread scheduler works in your host OS and your JVM, the nature of your CPU, how many cores in your CPU, and more.

Keep in mind that scheduling of threads is not free-of-cost. There is overhead in scheduling the the threads for their moments of execution. There is expense in changing between threads, the context switch. Hyper-threading is a hardware feature to lessen that cost of a context switch, but even then, switching between more than two threads for one core is back to a full context-switch.

So, it is naïve to think “the more the merrier” with threads. Quite often, you may find that fewer threads is more performant than too-many threads with too many context-switches.

Generally speaking, hundreds of active threads may be counter-productive, unless those threads spend most of their time doing nothing. And remember, your app (Tomcat + web apps) is not the only source of active threads on the host machine. The OS and other apps are likely running dozens of somewhat-active threads, and a few busier threads (local I/O, networking, and such).

For example, if you have a 4 core CPU with hyper-threading enabled, that means 8 logical cores, so you might expect to be using 5 or so for your dedicated Tomcat machine. If your threads were busy (CPU intensive) about a third of the time, then you might want to start with a thread pool of about 12-20. If the threads are busy only less than 5-10% of the time, then maybe pool of 100. Then monitor the real-world performance and see how it goes. If all the cores are blasting away at 100% utilization for minutes at a time, then you may be over-subscribing and might want to decrease the size of your thread-pool.

Regarding duration, if the threads are short-lived, but you may a great many of them during peak times of usage on your server, you may need to keep the pool smaller to avoid too many threads clamoring for the CPU simultaneously.

If you have many threads and each is very busy with the CPU such as encryption or codec, then you would want a thread pool size limited to less than the number of physical cores. For our example above, limit the pool to two or three of the four physical cores (8 logical hyper-threaded cores), leaving physical core(s) open for processes of the OS or other apps. Indeed, if you really have such very CPU-intensive tasks, you might consider disabling hyper-threading on your deployment computers. Hyper-threaded pairs of logical cores per physical core do not execute simultaneously, they trade-off back-and-forth with less cost for the context switch but not zero cost. If your tasks are extremely CPU-intensive (quite rare in common business apps), with no waiting-on-resource rest breaks, then there may be no benefit to hyper-threading.

Of course, you cannot know exactly the numbers discussed above as demanded by your particular web app(s) with your particular deployment. Trial-and-error is the only way to go.

Tip: Rather than hard-code your thread-pool sizes, you may wish to externalize the size of your thread pools to allow you to make changes at deployment-time. Perhaps set values to be retrieved through JNDI or some other external source.