Failed to stop the Elasticsearch thread in Tomcat

Question

I am using Tomcat 8.x for a Spring MVC-based website. The website calls Elasticsearch in the following way:

private static Client client;

Here is how I create the client object only once:

public static Client getClient() {
    if (client == null) {           
        Settings settings = Settings.settingsBuilder()
                .put("cluster.name", "my_cluster")
                .build();
                
        try {
            TransportAddress[] transportAddress = new TransportAddress[1];
            for (int k=0; k<adds.length; k++) {
                transportAddress[0] = new InetSocketTransportAddress(InetAddress.getByName('127.0.0.1'), 9300);
            }
            client = TransportClient.builder().settings(settings)
                    .addPlugin(DeleteByQueryPlugin.class)
                    .build()
                    .addTransportAddresses(transportAddress);
        
        } catch (UnknownHostException e) {
            logger.error("Failed to get ES client");;
        }           
    }       
    
    return client;
}

Here is how I use the client to access Elasticsearch:

public SearchResponse searchForMyPage(Long owerId, HttpServletRequest request, Account visitor, CampaignResultType type) {

    String query = 'this is my query string';
    
    if (client == null) {
        client = getClient();
    }
    
    SearchResponse sr = client.prepareSearch('MyIndex')
            .setTypes('MyDoc'))
            .setQuery(query)
            .setHighlighterNumOfFragments(1)
            .setSize(10).execute().actionGet();         
    return sr;
}

But I see many Elasticsearch related exceptions in Tomcat log. Something like the following:

19-May-2016 00:52:13.483 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [elasticsearch[Volpan][transport_client_worker][T#9]{New I/O worker #9}] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:

What is the correct way of using Elasticsearch in a Java web app?

Update

The following is the logs of Tomcat, show many Elasticsearch related errors, from the moment of Tomcat start to its shutdown.

20-May-2016 21:43:11.219 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [elasticsearch[War Machine][[timer]]] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 java.lang.Thread.sleep(Native Method)
 org.elasticsearch.threadpool.ThreadPool$EstimatedTimeThread.run(ThreadPool.java:703)
20-May-2016 21:43:11.219 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [elasticsearch[War Machine][scheduler][T#1]] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 java.lang.Thread.run(Thread.java:745)
20-May-2016 21:43:11.220 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [elasticsearch[War Machine][transport_client_worker][T#1]{New I/O worker #1}] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
 sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
 sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
 sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
 sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
 sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
 org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
 org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
 org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
 org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
 org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
 org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
 org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 java.lang.Thread.run(Thread.java:745)
20-May-2016 21:43:11.221 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [elasticsearch[War Machine][transport_client_worker][T#2]{New I/O worker #2}] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
 sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
 sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
 sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
 sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
 sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
 org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
 org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
 org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
 org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
 org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
 org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
 org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 java.lang.Thread.run(Thread.java:745)
20-May-2016 21:43:11.221 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [elasticsearch[War Machine][transport_client_worker][T#3]{New I/O worker #3}] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
 sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
 sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
 sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
 sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
 sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
 org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
 org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
 org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
 org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
 org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
 org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
 org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 java.lang.Thread.run(Thread.java:745)
20-May-2016 21:43:11.222 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [elasticsearch[War Machine][transport_client_worker][T#4]{New I/O worker #4}] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
 sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:296)
 sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:278)
 sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:159)
 sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
 sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
 org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:68)
 org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:434)
 org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:212)
 org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
 org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
 org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
 org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 java.lang.Thread.run(Thread.java:745)
20-May-2016 21:43:11.223 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [elasticsearch[War Machine][transport_client_worker][T#5]{New I/O worker #5}] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)

score 0 · Answer 1 · edited May 23 '17 at 11:52

0

You need to call client.close() when your application shuts down. How you do that depends on the specifics of your java web app.

For example (code copied from this stackoverflow answer), applicable for Servlet 3.0 spec:

@WebListener
public class BackgroundJobManager implements ServletContextListener {

    private ScheduledExecutorService scheduler;

    @Override
    public void contextInitialized(ServletContextEvent event) {
        scheduler = Executors.newSingleThreadScheduledExecutor();
        scheduler.scheduleAtFixedRate(new YourParsingJob(), 0, 5, TimeUnit.HOUR);
    }

    @Override
    public void contextDestroyed(ServletContextEvent event) {
        scheduler.shutdownNow();
    }

}

Or with Spring, in the class you create the client, define a @PreDestroy annotated method and call client.close() there:

@PreDestroy
public void destroy() {
  client.close();
}

edited May 23 '17 at 11:52

Community

1
1

answered May 20 '16 at 07:16

Andrei Stefan

51,654
6
98
89

Andrei, thanks for helping me! By "when your application shuts down", you mean shutting it down when a thread/web request is over? I hear that obtaining client is an expensive operation and so I made it static for reuse. Does `client.close` mean each time a new client has to be recreated? As you know, this is a web application. – curious1 May 20 '16 at 12:00
No, I mean when you shutdown Tomcat, because those messages refer to starting/stopping Tomcat. – Andrei Stefan May 20 '16 at 12:53
Andrei, I saw tons of these messages, not at the moment of Tomcat shutdown or start. It seems such message is thrown for each call to Elasticsearch. Thanks! – curious1 May 20 '16 at 13:38
Hm, are you sure your code is using the same Client instance? Doesn't sound like it does.... – Andrei Stefan May 20 '16 at 14:04
Andrei, you are right. The errors I have seems out of one failed stop and the program continues spew out error messages. – curious1 May 21 '16 at 01:51
Andrei, I updated with a segment of the log file. It appears the issue was caused by ONE failed stop of Elasticsearch thread, which generates error messages continuously. If I use your `public void destroy() {}` approach, what is the point of me having a static client object for reuse? Thanks! – curious1 May 21 '16 at 02:03

Failed to stop the Elasticsearch thread in Tomcat

Update

1 Answers1