What launches the Connection evictor thread and what can be done to avoid these threads to accumulate?

Question

My Java program runs under linux and indexes several directories (that are mounted via samba from different windows servers) with SOLR 7.4. It updates the different indexes (one index per indexed directory) one after the other and loops back infinitely.

While running on my dev machine I attached VisualVM to it and saw that the number of threads keeps increasing :

I understand from this post that it has something to do with memory leak (that I am also trying to find).

VisualVM shows that Connection evictor threads keep accumulating and are all in sleeping state :

But this post tells that sleeping threads won't add any load to the system (because they are idle), so they will not cause memory leak.

So my questions are :

Should I consider this behaviour as a problem, and if so where should I look at in the source code since I don't use http connection (which I read uses connection evictor) as all directories are locally mounted by the OS ?

Any help appreciated ;-)

In my application it is the Apache HttpClient that starts all these threads. It seems that for each connection a new so called evictor-thread is started. The purpose is to monitor for stale connections f.ex. if the server closed the socket. My application has a connection pool of max 50 connections, but still I see 1200 "connection evictor" threads. If I've understood the purpose of the evictor thread it should be max 50 of them, or? — perja, Aug 24 '20 at 18:17
Idle threads do consume memory for stack and thread-local storage. In a long run the growing horde of threads can consume all allocated memory. How do you create threads and how do you terminate them? Could you post the stack trace of a sleeping thread? What are they waiting for? — Pak Uula, Aug 26 '20 at 05:09
@PakUula @perja Thanks for your input. Actually I was creating a [`SolrClient` per core instead of per server](https://lucene.apache.org/solr/guide/7_1/using-solrj.html#base-urls). Now I reuse the same `SolrClient` throughout the application as recommended in the second post I quoted : `You should really be keeping one SolrClient per server node, and indicating which core to access with each request. One client object can access every core on a node. You do have to drop the core name from the URL. `. Now thread number remains around 20. — HelloWorld, Aug 28 '20 at 08:13

score 1 · Answer 1 · answered Sep 03 '20 at 04:37

To answer my own question briefly :

Yes having so many Connection Eviction threads is a problem on production since I assume it caused an OOM after a while.

For me the problem occured because I created too many SolrClients whereas a single one was needed (all the cores are on the same server).

So instead of

solrClient = new HttpSolrClient.Builder(
                getSolrHomePath() + "/" + getCoreName()
        ).
                build();

where getSolrHomePath() returns the path to solr-X.Y.Z/server/solr.

I now use

commonSolrClient = new HttpSolrClient.Builder(
                getSolrHomePath() 
        ).
                build();

and when I need to query a core I pass the core name as first parameter to the query function. Same for add/delete methods!

I had this issue in my java application that goes OutOffMemory after the jvm reaches the max thread. Also having several HttpSolrClient calling the solr server affect its performance — adia, May 29 '22 at 21:09

What launches the Connection evictor thread and what can be done to avoid these threads to accumulate?

1 Answers1