4

I have been reading up on how the settings of Spring's ThreadPoolTaskExecutor work together and how the thread pool and queue work. This stackoverflow answer as well as this and this article from Baeldung have been useful to me.

As far as I understand thus far, corePoolSize number of threads are kept alive at all time (assuming allowCoreThreadTimeOut is not set to true). If all of these threads are currently in use, any additional requests will be put on the queue. Once queueCapacity is reached, the thread pool size will be increased until maxPoolSize is reached.

Intuitively, I would have thought it would instead work as follows:
corePoolSize number of threads are kept alive at all time (again assuming allowCoreThreadTimeOut is not set to true). If all of these threads are currently in use and new requests come in, the pool size will be increased until maxPoolSize is reached. If there are then still more requests coming in, they will be put on the queue until queueCapacity is reached.

I wonder what would be the reasoning behind it working the way it is?

Christoph
  • 233
  • 1
  • 6
  • Let `corePoolSize = 4, maxPoolSize = 100, queueCapacity = 10`, then your assumption would mean that for 104 requests (which all are fired at once and take some time), 104 threads are created. The 105th will be then be put into the queue. And what happens after all the requests are done? Every thread is destroyed (except the ones from the core pool). Which is a lot of garbage created. And this will then happen everytime 104 requests are sent, so you're probably running into resource bottlenecks that way. The other way round your firstly filling the queue, ... – Lino Jul 14 '20 at 10:18
  • ... which is then worked by the core threads and *iff* the queue exceeds the maximum size, more threads are created, which all help then again to work the queue. That way you're not directly creating tens or even hundreds of threads, even though they may not be needed for long – Lino Jul 14 '20 at 10:19
  • Thank you @Lino, I understand, kind of. However, taking the same scenario as you suggested of 104 requests submitted simultaneously and each taking some time, the way it is actually implemented, the first 4 would execute immediately, the next 10 would go in the queue and as the next 90 requests come in, they would all have to pass through the queue and an additional 90 threads would be started up leaving the last 10 requests to remain in the queue... – Christoph Jul 14 '20 at 11:37
  • ...So we'd have 94 active threads and the last 10 waiting in the queue. As an aside, I actually think the 4 corePoolSize threads are included in the 100 maxPoolSize count. But anyway, the situation isn't much different. I think what I'm trying to say is that it would make more sense to me that once I've decided how many threads my system can maximally handle, this number of threads would first be exhausted before starting to queue requests. – Christoph Jul 14 '20 at 11:37
  • 2
    You're right, messed that a bit up. I think that the starting of new threads could potentially take longer than simply queing them. So having a large queue size, a moderate maxPoolSize and a rather small corePoolSize would probably work the best. – Lino Jul 14 '20 at 11:43
  • The problem is that say I would like to have at most 50 threads active at any time and if more requests come in, I'd like to queue a large number of requests before failing. Currently, my only option is to set corePoolSize and maxPoolSize to 50 and then a large queue size. But this means that all 50 threads are always active, even if they're all idle. It would be more useful to me if it worked like I describe and then to set corePoolSize to say 10, maxPoolSize to 50 and then a large queue size to handle any additional incoming requests. – Christoph Jul 14 '20 at 11:58
  • I think it would be nice to have both options available, that would be really neat! – Christoph Jul 14 '20 at 12:00
  • 1
    I think you may want to have a look at `setAllowCoreThreadTimeOut(boolean)` which allows the core threads to time out and be destroyed again – Lino Jul 14 '20 at 12:17
  • Yes, that is an option, though then I cannot keep any of the threads alive and ready at all times... – Christoph Jul 14 '20 at 12:36
  • Christoph, I've looked a bit through the code of the class, it seems that the `ThreadPoolExecutor` will shrink the pool size back to corePoolSize if the threads are idle, with and without the before mentioned flag set to true – Lino Jul 14 '20 at 13:15

1 Answers1

2

The first reference you should check is the documentation.

Right from the documentation for ThreadPoolExecutor (ThreadPoolTaskExecutor is "just" a wrapper):

A ThreadPoolExecutor will automatically adjust the pool size (see getPoolSize()) according to the bounds set by corePoolSize (see getCorePoolSize()) and maximumPoolSize (see getMaximumPoolSize()). When a new task is submitted in method execute(Runnable), if fewer than corePoolSize threads are running, a new thread is created to handle the request, even if other worker threads are idle. Else if fewer than maximumPoolSize threads are running, a new thread will be created to handle the request only if the queue is full. [...]

If the pool currently has more than corePoolSize threads, excess threads will be terminated if they have been idle for more than the keepAliveTime (see getKeepAliveTime(TimeUnit)). This provides a means of reducing resource consumption when the pool is not being actively used. If the pool becomes more active later, new threads will be constructed. [...]

(You haven't mentioned the parameter for the BlockingQueue but I suggest you to read about it as well. It's very interesting.)

Why do the parameters not work like you've suggested they should?

If the pool size would be increased up to maximumPoolSize before tasks are queued (like you've proposed), you'd have one problem: You'd have removed the thread pool's ability to determine when a new worker is worth it.

The corePoolSize is the amount of workers that stay in the pool. The benefit is that you don't have to create, terminate, create, terminate, create ... new workers for a given workload. If you can determine how much work there will always be, it's a smart idea to set the corePoolSize accordingly.

The maximumPoolSize determines the maximum amount of workers in the pool. You want to have control over that as you could have multiple thread pools, hardware restrictions or just a specific program where you don't need as many workers.

Now why does the work queue get filled up first? Because the queue capacity is an indicator for when the amount of work is so high, that it's worth it to create new workers. As long the queue is not full, the core workers are supposed to be enough to handle the given work. If the capacity is reached, then new workers are created to handle further work.

With this mechanism the thread pool dynamically creates workers when there is a need for them and only keeps so many workers as there is usually need for. This is the point of a thread pool.

akuzminykh
  • 4,522
  • 4
  • 15
  • 36
  • Thank you for the detailed response @akuzminykh. I still think that it would work just fine with the way I suggest. If the corePoolSize is exhausted, that is the indication that the amount of work is so high that it's worth to create new workers. I think I can kind of see the point of the queue as implemented for the scenario where each thread is in use for a short time. In this case, as requests come in, the core pool together with the queue can be enough to handle a relatively high volume of incoming requests without needing to start additional workers... – Christoph Jul 15 '20 at 10:26
  • ... However, in the case where each request keeps its threads in use for a longer period of time, the queue in the middle doesn't help much. And in this scenario, there is no way for me to make use of the max pool size to handle incoming requests, while at the same time having a large queue so as not to reject additional incoming requests. Maybe my suggestion would be that it would be most useful to have both queues. The one as currently implemented and the other as I suggest. – Christoph Jul 15 '20 at 10:30
  • @Christoph I haven't said that it won't work. I've said that the mechanism you've proposed will remove the thread pool's ability to determine when a new worker is needed. There are cases where you want that and in fact this is what `Executors.newCachedThreadPool()` gives you: A thread pool that has no queue size and instantly creates new threads on upcoming tasks. This is due to the used `BlockingQueue`, the `SynchronousQueue`, and the used parameters `corePoolSize = 0` and `maximumPoolSize = Integer.MAX_VALUE`. You can configure thread pools to work exactly like you've proposed. ... – akuzminykh Jul 15 '20 at 11:10
  • @Christoph ... The implementation as it is is very flexible. What you've proposed is just one specific configuration. Check out the documentation; there are many settings you can make. Check out the different `BlockingQueue`s you can use, like I've already suggested. The important thing is that implementing thread pools like you've described in *general* will add restrictions. You won't be able to configure a thread pool that can determine when new workers are truly needed without complicated workarounds. The current implementation is simple, effective and highly customizable. – akuzminykh Jul 15 '20 at 11:10
  • @Christoph Feel free to ask if there are still things unclear. But please don't make this an extended discussion, which is what the comment section is not for. If you feel like there is something fundamentally unclear or wrong, you may want to post a new question and specify that. I'm very sure that my answer and the additional comments answer the current question very well. – akuzminykh Jul 15 '20 at 11:25
  • you're right, you do answer the current question. And I don't want to continue a long discussion here. Thus my final comment: you are right, the implementation is rather flexible. But it has limitations. Even with all its flexibility I cannot have a bounded reasonably sized maxPoolSize which actually gets fully utilized and then have a large queue to queue any additional requests on top of that, which in my mind is a reasonable requirement. My original proposal has other limitations as you've pointed out. Thus I would suggest to have 2 queues. This would be the most flexible. – Christoph Jul 15 '20 at 12:33