Java leaks memory out of heap when releasing threads

Question

I have a small application that allocates some 25000 threads and then releases them. While the threads are being released the memory consumption of the application rises and remains high even after all of the threads exit.

top looks like this:

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
9133 root      20   0 22.601g 8.612g  12080 S   0.0  9.1   1:18.61 java

while jmap -heap looks like this:

Attaching to process ID 9133, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.66-b17

using thread-local object allocation.
Parallel GC with 18 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 40
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 1073741824 (1024.0MB)
   NewSize                  = 104857600 (100.0MB)
   MaxNewSize               = 104857600 (100.0MB)
   OldSize                  = 968884224 (924.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 1073741824 (1024.0MB)
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 78643200 (75.0MB)
   used     = 1572864 (1.5MB)
   free     = 77070336 (73.5MB)
   2.0% used
From Space:
   capacity = 13107200 (12.5MB)
   used     = 0 (0.0MB)
   free     = 13107200 (12.5MB)
   0.0% used
To Space:
   capacity = 13107200 (12.5MB)
   used     = 0 (0.0MB)
   free     = 13107200 (12.5MB)
   0.0% used
PS Old Generation
   capacity = 968884224 (924.0MB)
   used     = 1264416 (1.205841064453125MB)
   free     = 967619808 (922.7941589355469MB)
   0.13050227970271916% used

808 interned Strings occupying 54648 bytes.

As far as I can see, there is nothing in the jmap report that can explain the 8.612g reported by top.

Java version is oracle 1.8.0_66

The application runs on Red Hat Enterprise Linux Server release 7.1 (Maipo).

The code of the application is below:

import java.util.*;
import java.util.concurrent.atomic.AtomicInteger;

public class WaitTest {
    static AtomicInteger ai = new AtomicInteger(0);

    public static void main(String[] args) {
        List<WaitingThread> threads = new LinkedList<>();
        while (true) {
            System.out.println("number of threads: " + ai.get());
            String s = System.console().readLine();
            if (s == null)
                System.exit(0);
            s = s.trim();
            if (s.isEmpty())
                continue;
            char command = s.charAt(0);
            if (command != '+' && command != '-') {
                System.out.println("+ or - please");
                continue;
            }
            String num = s.substring(1);
            int iNum;
            try {
                iNum = Integer.parseInt(num.trim());
            } catch (Exception ex) {
                System.out.println("valid number please");
                continue;
            }
            if (command == '+') {
                for (int i = 0; i < iNum; i++) {
                    WaitingThread t = new WaitingThread();
                    t.start();
                    threads.add(t);
                }
            }
            if (command == '-') {
                Set<WaitingThread> threadsToJoin = new HashSet<>();
                for (Iterator<WaitingThread> it = threads.iterator(); it.hasNext(); ) {
                    if (iNum > 0) {
                        WaitingThread t = it.next();
                        threadsToJoin.add(t);

                        synchronized (t.lock) {
                            t.lock.notify();
                        }

                        it.remove();
                        iNum--;
                    } else
                        break;

                }
                for (WaitingThread t : threadsToJoin)
                    try {
                        t.join();
                    } catch (InterruptedException e) {
                        e.printStackTrace();
                    }
            }
            System.gc();
        }
    }

    static class WaitingThread extends Thread {
        public final Object lock = new Object();

        @Override
        public void run() {
            ai.incrementAndGet();
            try {
                deepStack(200);
            } catch (InterruptedException ex) {
            } catch (Exception ex) {
                System.out.println("exception in thread " + Thread.currentThread().getName() + ", " + ex.getMessage());
            } finally {
                ai.decrementAndGet();
            }
        }

        private void deepStack(int depth) throws InterruptedException {
            if (depth == 0) {
                synchronized (lock) {
                    lock.wait();
                }

            } else
                deepStack(depth - 1);
        }
    }

}

Here is a link to pmap output:

Here is a link to jmap -dump (hprof)

Answer 1

Don't all the threads first get created during the command "+" in the list variable 'threads' which will never go out of scope? That means the linked list is holding references to all the threads and won't be garbage collected until Main() exits. Also, it is possible that the LinkedList itself is leaking memory. Even if that isn't the issue you should probably consider using an ArrayList instead of a LinkedList.

Also, here is another question that asks about finding memory leaks in Java.

Answer 2

The application memory is only loosely coupled to a heap size. When the GC runs and cleans things up, it cleans up the heap, not the system memory. In fact the JVM may not even make the system call to reduce system memory even though it frequently frees up heap space.

Answer 3

Each thread need a stack (allocated memory) which is not part of the java heap.

The default size of the stack depends of the jvm version, the os, etc..

You can tune it with the -Xss jvm parameter.

25000 threads can easily explain your 8.6g

Update following Amir comment :

Not 100% sure but looking at tour pmap log, you probably have a problem with the arena allocator in glibc

See the all the 64Mb allocations

Check that your glibc version is > 2.10

Try to run your program with the environment variable MALLOC_ARENA_MAX=2 set. You can play with the actual value.