New threads are not created by the Java ThreadPoolTaskExecutor

Question

This particular Java application uses the following dependency and service versions.

• Java 7
• Tomcat 7
• Spring 4.3.6

It has the following method.

public void submitSms(final SmppMoSMS smppMoSMS) {
    logger.info("** SMS received from SMSC [{}]", smppMoSMS.toString());
    int i = Thread.activeCount();
    logger.info("Point 1: Active thread count : [{}]", i);
    taskExecutor.execute(new Runnable() {
        @Override
        public void run() {
            try {
                logger.info("Thread start ");
                logger.debug("SMS routing to URL [{}] [{}]", smppMoSMS.getOperator(), smppMoSMS.getApplicationUrl());
                logger.info("SMS routing to URL [{}] [{}]", smppMoSMS.getOperator(), smppMoSMS.getApplicationUrl());
                if (smppMoSMS.getApplicationUrl() == null) {
                    return;
                }
                HttpEntity<SmppMoSMS> entity = new HttpEntity<>(smppMoSMS, headers);
                ResponseEntity<String> response = restTemplate.exchange(smppMoSMS.getApplicationUrl(), HttpMethod.POST, entity, String.class);
                if (HttpStatus.OK == response.getStatusCode()) {
                    logger.info("SMS sending success [{}] [{}] [{}]", smppMoSMS.getOperator(), smppMoSMS.getApplicationUrl(), 200);
                } else {
                    logger.info("SMS sending fail [{}] [{}] [{}]", smppMoSMS.getOperator(), smppMoSMS.getApplicationUrl(), response.getStatusCode());
                }
            } catch (Throwable e) {
                logger.error("SMS sending fail [{}] [{}]", smppMoSMS.getOperator(), smppMoSMS.getApplicationUrl());
                logger.error("SMS ROUTING FAIL", e);
            }
            logger.info("Thread stop ");
        }
    });
    logger.info("Point 2: Active thread count : [{}]", i);
    MDC.clear();
}

Related to the TaskExecutor following bean configuration is available:

<bean id="threadPool" class="org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor"
      destroy-method="destroy">
    <property name="corePoolSize" value="5"/>
    <property name="maxPoolSize" value="200"/>
    <property name="queueCapacity" value="10000"/>
    <property name="allowCoreThreadTimeOut" value="true"/>
    <property name="waitForTasksToCompleteOnShutdown" value="true"/>
</bean>

Related to the RestTemplate the following bean configurations are available.

<bean id="restTemplate" class="org.springframework.web.client.RestTemplate">
    <constructor-arg>
        <bean class="org.springframework.http.client.HttpComponentsClientHttpRequestFactory">
            <property name="readTimeout" value="30000"/>
            <property name="connectTimeout" value="30000"/>
        </bean>
    </constructor-arg>
</bean>

The high-level issue observation is described below.

1. Let's say the application is started

2. Application is functioning as expected including the following operations

   • HTTP calls are generated by the restTemplate according to the Tcpdumps and the REST API server logs
   • All logs are printed on log files

3. After 2-3 hours suddenly we can observe the following behaviours of the application.

   • HTTP calls are not generated by the restTemplate according to the TcpDumps and the REST API server logs
   • Only the following logs are available

2022:10:01 07:17:10.623 INFO [correlationId=1664588830622, sequence=2244] ** SMS received from SMSC [SmppMoSMS{message='Test by developer ', senderAddress='94XXXXXXX', recipientAddress='94XXXXX', encoding='0', operator='XXXXXXXX', ncsType='null', correlationId='1664588830622'}]

2022:10:01 07:17:10.623 INFO [correlationId=1664588830622, sequence=2244] Active thread count : [114]

2022:10:01 07:17:10.623 INFO [correlationId=1664588830622, sequence=2244] Thread stop

2022:10:01 07:17:10.623 INFO [correlationId=1664588830622, sequence=2244] Active thread count : [114]

This "Active thread count" doesn't increase after this.

All the things are getting fine by just an application restart or the whole tomcat server restart. NOTE: This application was in production for several years and we get this issue from the last 2 months only without any code or configuration changes.

I assume new threads are not created even though the configuration allows up to 200 threads.

1. What can be the reasons for the above assumptions or the observations?
2. Any other assumptions?
3. How can I further investigate the issue?
4. Any suggestions to fix the issue?

Answer 1

What can be the reasons for the above assumptions or the observations?

One of the issues that you need to realize is that you will not get more than the core number of threads allocated in your thread-pool until the queue fills up. I kid you not. In your case, it's only when the 5 threads are busy and 10000 jobs are queued that the 6th thread will be started.

I would set the corePoolSize and the maxPoolSize both to be 100 and then set the keepAliveSeconds to 30 or something if you want the core threads to drop down during periods of inactivity. See this question for more details: How to get the ThreadPoolExecutor to increase threads to max before queueing?

Couple other comments:

int i = Thread.activeCount();
logger.info("Point 1: Active thread count : [{}]", i);
askExecutor.execute(new Runnable() { ... });
logger.info("Point 2: Active thread count : [{}]", i);

The active-count won't change at point 2 since you aren't making another call to Thread.activeCount(). Also, you are scheduling a job to be run by the thread-pool but that doesn't necessarily mean that a thread will be created.

If you are using a ThreadPoolExecutor directly (as opposed to the Spring wrapper) then you could call pool.getActiveCount() and other methods to get a more accurate picture of the threads running in your thread pool. The 114 threads running in your application may be associated with the tomcat handlers and other JVM threads instead of your pool specifically. You can also use jconsole, jstack, or a SIGQUIT to dump the thread stacks and see how many threads are associated with your pool. The thread names should be something like xxx-pool-1 or something unless you are using a thread-factory.

Lastly a general comment. Maybe this is obvious but you really should be trying to optimize your throughput as opposed to worrying about the number of threads. 100 may be too many threads if you are pounding the server too hard or if the job is computational or 100 may be too few if your HTTP connections take a while to be satisfied because of latency.