2

Doing a thread dump on a highly loaded application with CPU, I see a lot of threads in this state:

"ajp-executor-threads - XXXXXX" prio=10 tid=0x00002b04b8b33801 nid=0x5327 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE

What is really strange to me is that there is no stacktrace at all and that total number of ajp-thread is higher than max-threads (below) configured

It is happening with an application running on:

  • JBoss 7
  • Java 7u75
  • Redhat 5.11
  • Running on VMWare Enterprise / vSphere 5.5

Configuration of executor is:

 <subsystem xmlns="urn:jboss:domain:threads:1.1">
        <bounded-queue-thread-pool name="ajp-executor">
            <core-threads count="32"/>
            <queue-length count="1"/>
            <max-threads count="300"/>
            <keepalive-time time="5" unit="seconds"/>
        </bounded-queue-thread-pool>
    </subsystem>

Note that load is very high as on this host:

  • CPU reaches 70%

  • Load is at 4 (== number of vCPU)

Note these threads are not idle threads as an idle thread has this stack trace:

 "Reference Handler" daemon prio=5 tid=0x00007f92cb00e800 nid=0x3703 in Object.wait() [0x000000012057e000]
     java.lang.Thread.State: WAITING (on object monitor)
     at java.lang.Object.wait(Native Method)
     - waiting on <0x00000007aaa84470> (a java.lang.ref.Reference$Lock)
     at java.lang.Object.wait(Object.java:503)
     at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
     - locked <0x00000007aaa84470> (a java.lang.ref.Reference$Lock)
pmpm
  • 705
  • 1
  • 5
  • 20

2 Answers2

3

After further analysis I found that issue is due to remote debugging being enabled through:

-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=XXXXXXX

This explains these strange empty stack traces in thread dump.

pmpm
  • 705
  • 1
  • 5
  • 20
  • Thanks. Debugger went unnoticed and was causing huge thread issues for me kind of like thread leak. Your answer helped. – vignesh Jun 29 '22 at 11:44
-3

Those are your idle threads in the AJP executor connection pool. You have set the core threads to 32 meaning that the pool will always maintain 32 threads in the connection pool, although they may be idle. With your configuration you could see up to 300 threads but any thread over 32 will only wait 5 seconds before dying and being removed from the connection pool.

As for you CPU load, I doubt its in any way related to these idle connection pool threads.

See John Skeet's answer here for some more info on connection pools and keep alive: https://stackoverflow.com/a/10379348/91866

Community
  • 1
  • 1
Ian Dallas
  • 12,451
  • 19
  • 58
  • 82
  • 1
    I don't think idle threads in the AJP Executor connection pool have this kind of stacktrace – pmpm Jul 02 '15 at 19:57
  • Did you post a stack trace? You posted a dump of the threads you had. Idle threads wouldn't have a stack trace yet so not really sure what you were looking for. Was my answer inaccurate given your question? The thread you listed is an idle ajp executor thread and you would see a lot of them given your configuration. If you were expecting to someone to just triage your performance problem then your question was very unclear and likely would not be answered here on StackOverflow. – Ian Dallas Jul 02 '15 at 20:13
  • first thanks for your answer. Then sorry but in my question , I posted a stacktrace from the thread dump. And I confirm to you it is not an idle thread. I posted question because I had really no idea after investigating for some time and then I found by myself and shared the answer. Anyway thanks again and sorry for not saying so the first time. – pmpm Jul 02 '15 at 20:51
  • I downvoted your answer as it is wrong to me in this particular case, but I upvoted some of your answers that look ok to me. Have a good day – pmpm Jul 02 '15 at 21:01