Is there thread depletion risk when multithreading calls to cache in a servlet?

Question

I have the following code, abbreviated here:

 public final void doPost(HttpServletRequest request,
     HttpServletResponse response) {

            int itemCount = itemsToGetFromCache.size();
            ExecutorService service = null;
           List<Future<?>> futures = null;
            service = Executors.newFixedThreadPool(itemCount);
           futures = new ArrayList<Future<?>>();
           for (int i=0; i<itemCount; i++)
           {
                final int j = i;
                    Future<?> f = service.submit(new Callable<Void>() {

                        @Override
                        public Void call() throws Exception {
                            getItemFromRemoteCacheIfAvailableAndStoreInMemory(itemsToGetFromCache(j));
                            return null;
                        }});
                    futures.add(f);
           }
            // wait for all tasks to complete before continuing
            for (Future<?> f : futures)
            {
               try {
                   f.get();
                } catch (Exception e) {
                    //handle exception
                }
            }

}

It's running in a Tomcat 7. The typical item size is 30 and typical simultaneous users is 200. Some have warned that this could cause the server threads to be maxed out and connections to be denied. Note, that the calls to the remove cache will be brief most often, taking around 60 millis.

Basically, I'm just trying to make the calls to cache faster by running them in parallel. If the results are not in cache, they'll be pulled from a database and cached subsequently.

Is there a problem here? I don't think the server connection pool size is linked to the maximum number of threads the server can handle when spawning them this way. Am I right in assuming this? Are there any other concerns?

Ramesh PVK · Answer 1 · 2015-07-09T20:54:30.717

2

You should avoid creating threadpool for every request. Instead, you can use common thread pool.

Again, 6000(200*30) threads might actually degrade the performance because of rigorous thread context switching.

10 threads can perform better than 6000 threads. How man threads you can use depends on various factors such as :

number of cores
the amount of blocking within each thread event.

For Ex, if there is no I/O blocking in the code being executed such as database connections, etc. The number of threads must be just same as the number of cores. If there is some amount of blocking you should measure how many more threads you can have to keep the CPU's busy when some thread is getting blocked.

It is very difficult to guess how many threads to be used. You have to measure it how the program is performing for various thread pool size. Be it 10 or 100 or 200. Make it configurable. Definitely , it is bad to have many threads.

edited Jul 09 '15 at 20:54

answered Jul 09 '15 at 20:11

Ramesh PVK

15,200
2
46
50

I think that's the key point, there definitely is some blocking as each thread run in parallel has to wait for the network cache call to return. So you have essentially 30 threads doing nothing but waiting. They aren't using CPU, so why is that a problem other than context switches? – Jul 09 '15 at 20:33
1) The problem is context switching. 2) Creating a thread has overhead. Creating it every time is also an issue. – Ramesh PVK Jul 09 '15 at 20:35
I'll have to figure out then if the cost of the 30 context switches is greater than the gain of 30*60millis per call. – Jul 09 '15 at 21:48

score 0 · Accepted Answer · answered Jul 09 '15 at 20:12

You are right that the servers request thread pool is not being used and will not be maxed.

However will this result in a lot of threads being created and too many context switchs - so you are really not getting any benefit from creating the ExecutorService per request. This may overwhelm the server and result in connections to be denied since the server is busy

score 0 · Answer 3 · edited May 23 '17 at 10:26

Every reasonable application server will have a request processing pool, which usually has a maximum number of threads. Your code delegates part of request processing to newly created threads, which are not from the request processing thread pool, so the limit of that pool is not hit any sooner. In fact, because the request processing takes less time, the request processing threads are returned to the pool faster, increasing the number of requests per second the request processing thread pool can handle.

Is there some other limit for the number of threads? As Java uses operating system threads, this is typically up to the operating system, and you may find How many threads can a Java VM support? interesting reading. However, a couple thousand threads are certainly possible, as the following test program (which I have adapted from the linked question) demonstrates:

public class Test {

    private static final AtomicInteger count = new AtomicInteger();

    public static void main(String[] argv) {
        for (;;) {
            new Thread() {
                public void run() {
                    count.incrementAndGet();
                    System.err.println("New thread #" + count + " has started");
                    for (;;) {
                        try {
                            Thread.sleep(1000);
                        } catch (InterruptedException e) {
                            throw new RuntimeException(e);
                        }
                    }
                }
            }.start();
        }
    }
}

On my Windows 8.1 workstation, its output ends with:

New thread #149932 has started
New thread #149934 has started
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000067242357, pid=4324, tid=606772
#
# JRE version: Java(TM) SE Runtime Environment (8.0_05-b13) (build 1.8.0_05-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.5-b02 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# V  [jvm.dll+0x1f2357]Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
    at java.lang.Thread.start0(Native Method)
    at java.lang.Thread.start(Thread.java:714)
    at stackoverflow.Test.main(Test.java:23)

i.e. we get a JVM crash - at about 150000 threads. 6000 threads are probably fine on all platforms.

However, you don't invoke shutdown() after you are done with the executor, causing the threads to continue to wait for more tasks - presumably until the executor is garbage collected. This is a needless waste of memory. Moreover, on old JVMs were thread stacks are in a separate memory space, you may run out of thread stack space before the garbage collector runs on the heap and collects the executor object. This may be what your colleagues were referring to.

I find it questionable that you do not reuse threads - after all, creating a thread involves some overhead (such as allocating space for its stack). Also, a shared thread pool would allow you to limit the number of concurrent cache accesses (if talking to the cache involves network I/O, 6000 open tcp sockets may exhaust file handles, or simply overload the other system). The easiest way for reusing threads is to reuse the executor.

Is there thread depletion risk when multithreading calls to cache in a servlet?

3 Answers3