why got different number of threads?

Question

Part of my scala program:

import java.util.concurrent.{ExecutorService, Executors, TimeUnit}

val genericExecutorService = Executors.newCachedThreadPool()

val scheduledExecutorService = Executors.newScheduledThreadPool(4)

val scheduledExecutorService1 = Executors.newScheduledThreadPool(1)

val scheduledExecutorService2 = Executors.newScheduledThreadPool(1)

I counted the number of threads from the log file, and the number is 120. Part of log file:

2018-10-18 00:00:00,421 INFO  [pool-7-thread-32] xxx

Count the thread in log file:

$ grep -r "thread" xxx.log | grep -Po '(?<=(\[)).*(?=\])' | sort | uniq -c | wc -l
120

I counted the threads of java by

$ cat /proc/2966/status | grep Threads
Threads:    1524
$ ps -eLF| grep -c java
1525
$ ps -L -o pid= -p 2966 | wc -l
1524

Why 1525 and 120 so different? Any hints welcomed. Thanks.

Answer 1

You can use better tool to analyse the threads in JVM application instead of relying on logging. There are visual tools like jmc and jconsole. There is jstack if you on PROD machine where visual tools may not be available.

If I've following JVM application,

object Testing {

  def main(args: Array[String]): Unit = {

    import java.util.concurrent.{ExecutorService, Executors, TimeUnit}

    val genericExecutorService = Executors.newCachedThreadPool()

    val scheduledExecutorService = Executors.newScheduledThreadPool(4)

    val scheduledExecutorService1 = Executors.newScheduledThreadPool(1)

    val scheduledExecutorService2 = Executors.newScheduledThreadPool(1)

    genericExecutorService.execute(() => {
      println("0")
    })

    scheduledExecutorService.execute(() => {
      println("1")
    })

    scheduledExecutorService1.execute(() => {
      println("2")
    })

    scheduledExecutorService2.execute(() => {
      println("3")
    })
  }

}

I can get the threads using jstack using the proccessId of that application,

$ jstack 68209 | grep thread
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.151-b12 mixed mode):
"pool-4-thread-1" #14 prio=5 os_prio=31 tid=0x00007ff1b812a800 nid=0xa503 waiting on condition [0x000070000c305000]
"pool-3-thread-1" #13 prio=5 os_prio=31 tid=0x00007ff1b812a000 nid=0xa703 waiting on condition [0x000070000c202000]
"pool-2-thread-1" #12 prio=5 os_prio=31 tid=0x00007ff1b80f9000 nid=0xa903 waiting on condition [0x000070000c0ff000]
"pool-1-thread-1" #11 prio=5 os_prio=31 tid=0x00007ff1b9869800 nid=0x5803 waiting on condition [0x000070000bffc000]
"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007ff1b9802800 nid=0x1c07 runnable 
"GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007ff1b9001000 nid=0x1d03 runnable 
"GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007ff1ba804800 nid=0x2b03 runnable 
"GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007ff1ba805000 nid=0x5303 runnable 
"GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007ff1ba805800 nid=0x5103 runnable 
"GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007ff1b8810800 nid=0x2c03 runnable 
"GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007ff1b981e800 nid=0x4e03 runnable 
"GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007ff1b981f000 nid=0x4c03 runnable 

you can simply do jstack pid as well but that gives whole dump of a thread including thread_state and all.

Using jmc visual tool,

Might be helpful: Monitoring queue of ExecutionContextExecutor “scala.concurrent.ExecutionContext.Implicits.global”