-XX:NativeMemoryTracking not picked up from _JAVA_OPTIONS

Question

I'm not able to pick up NMT settings via _JAVA_OPTIONS. My environment is a Docker container on an kubernetes cluster with _JAVA_OPTIONS containing a flag to enable NMT (see below).

To demonstrate I've created a little demo class Hello which prints a line once a second.

a) I run Hello without command line args:

[root@ticc-services-abo-3354894200-5fixb tmp]# echo $_JAVA_OPTIONS
-Xmx512m -Xms512m -XX:NativeMemoryTracking=summary
[root@ticc-services-abo-3354894200-5fixb tmp]# java Hello
Picked up _JAVA_OPTIONS: -Xmx512m -Xms512m -XX:NativeMemoryTracking=summary
hello 0
hello 1
...

On another console I try to get an NTM summary, but it's not accessible:

[root@ticc-services-abo-3354894200-5fixb /]# jcmd
Picked up _JAVA_OPTIONS: -Xmx512m -Xms512m -XX:NativeMemoryTracking=summary
1 /opt/abo.jar
740 Hello
766 sun.tools.jcmd.JCmd
[root@ticc-services-abo-3354894200-5fixb /]# jcmd 740 VM.native_memory
Picked up _JAVA_OPTIONS: -Xmx512m -Xms512m -XX:NativeMemoryTracking=summary
740:
Native memory tracking is not enabled

b) Now I set the NMT flag directly on the command line:

[root@ticc-services-abo-3354894200-5fixb tmp]# java -XX:NativeMemoryTracking=summary Hello
Picked up _JAVA_OPTIONS: -Xmx512m -Xms512m -XX:NativeMemoryTracking=summary
hello 0
hello 1
...

and NMT works:

[root@ticc-services-abo-3354894200-5fixb tmp]# jcmd
Picked up _JAVA_OPTIONS: -Xmx512m -Xms512m -XX:NativeMemoryTracking=summary
1 /opt/abo.jar
834 Hello
846 sun.tools.jcmd.JCmd
[root@ticc-services-abo-3354894200-5fixb tmp]# jcmd 834 VM.native_memory summary
Picked up _JAVA_OPTIONS: -Xmx512m -Xms512m -XX:NativeMemoryTracking=summary
834:

Native Memory Tracking:

Total: reserved=1873866KB, committed=574898KB
-                 Java Heap (reserved=524288KB, committed=524288KB)
                        (mmap: reserved=524288KB, committed=524288KB)
...

After modifying Hello to print its heap total and max, we see that -Xmsand -Xmxare picked up but the NMT flag not, since Java does not complain about the bad NMT flag:

[root@ticc-services-abo-3354894200-5fixb tmp]# echo $_JAVA_OPTIONS
-Xmx256m -Xms128m -XX:NativeMemoryTracking=summary_FOO
[root@ticc-services-abo-3354894200-5fixb tmp]# java Hello
Picked up _JAVA_OPTIONS: -Xmx256m -Xms128m  -XX:NativeMemoryTracking=summary_FOO
heap: total=128974848, max=239075328
hello 0
hello 1
hello 2

Here is my environment:

[root@ticc-services-abo-3354894200-5fixb /]# uname -a
Linux ticc-services-abo-3354894200-5fixb 4.7.3-coreos-r2 #1 SMP Tue Nov 1 01:38:43 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
[root@ticc-services-abo-3354894200-5fixb /]# java -version
Picked up _JAVA_OPTIONS: -Xmx512m -Xms512m -XX:NativeMemoryTracking=summary
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

This looks quite strange, so any hints or ideas are very welcome.

I may be missing something, but where does the "Picked up..." line come from? — RealSkeptic, Jan 02 '17 at 17:07
This should be a standard into from the JVM to `stderr` about reading `_JAVA_OPTIONS` — Udo Bellack, Jan 02 '17 at 17:09
Standard, it isn't. It is not documented anywhere on the Oracle website. There is no way to know which options it accepts and which it doesn't. — RealSkeptic, Jan 02 '17 at 17:24
@UdoBellack It looks to be a bug in `jcmd` Maybe create a JIRA or post it in JDK Forum. Even I faced the same issue and I tried both `_JAVA_OPTIONS` and `JAVA_TOOL_OPTIONS`. — Kishore Bandi, Jan 03 '17 at 08:04
@BandiKishore You're right, `-XX:NativeMemoryTracking=...` is neither picked up from `JAVA_TOOL_OPTIONS` nor `_JAVA_OPTIONS` despite the value set there is reported via `-XX:+PrintFlagsFinal`. What a mess! — Udo Bellack, Jan 03 '17 at 09:04
My solution was to put this option as a process argument, e.g. command line option. (Since I had to start a docker image from kubernetes, it was rather a process arg than a command line option) — Udo Bellack, Oct 29 '17 at 14:51

Mario.Cadiz · Answer 1 · 2019-12-14T15:44:19.580

I do believe there is a confusion between an environment variable existent in your machine and the VM arguments to start this Java process.

You can call it _JAVA_OPTIONS, JAVA_OPTIONS, MY_STARTUP_OPTION or any other name, there is nothing special with that name, it's not a reserved word for the Java process.

The fact that your Java code is printing out the value of that variable doesn't make it special either:

  System.out.println("Picked up JAVA_OPTIONS: "+System.getenv("_JAVA_OPTIONS"));
  System.out.println("Picked up JAVA_OPTIONS: "+System.getenv("JAVA_OPTIONS"));

For this Java process, it is just another common env variable. Here the important is knowing if that variable is used to startup your process and based on the sequence of steps shown in this example, the answer is no.

If you wish to make that variable is used to startup that Java process, you should use:

java $JAVA_OPTIONS Hello

or

java $_JAVA_OPTIONS Hello

Additionally, to print out the VM args for this Java process, you can use:

ManagementFactory.getRuntimeMXBean().getInputArguments();

Here is the complete Java code

import java.lang.management.ManagementFactory;
import java.util.List;

public class Hello {

    public static void main(String[] args) {
        int i = 0;
        System.out.println("Picked up JAVA_OPTIONS: "+System.getenv("JAVA_OPTIONS"));
        System.out.println("VMArguments: "+Hello.vmArguments().toString());
        try {

            while (true) {
                System.out.println("hello " + (i++));
                Thread.sleep(1000);
            }
        } catch (InterruptedException e) {
            System.out.println("Good bye");
            e.printStackTrace();
        }

    }

    static List<String> vmArguments() {
        return ManagementFactory.getRuntimeMXBean().getInputArguments();
      }

}

And an example of Execution:

C:\Hello\bin>echo %JAVA_OPTIONS%
-Xmx512m -Xms512m -XX:NativeMemoryTracking=summary

C:\Hello\bin>echo %MY_STARTUP_OPTION%
-XX:NativeMemoryTracking=detail

C:\Hello\bin>java %MY_STARTUP_OPTION% Hello
Picked up JAVA_OPTIONS: -Xmx512m -Xms512m -XX:NativeMemoryTracking=summary
VMArguments: [-XX:NativeMemoryTracking=detail]
hello 0

C:\Hello\bin>java %JAVA_OPTIONS% Hello
Picked up JAVA_OPTIONS: -Xmx512m -Xms512m -XX:NativeMemoryTracking=summary
VMArguments: [-Xmx512m, -Xms512m, -XX:NativeMemoryTracking=summary]
hello 0

On another terminal you can use $JAVA_HOME/bin/jps or ps -fea | grep java. Here is an example in Windows:

C:\Users\MyUser>jps -lv
5644 Hello -Xmx512m -Xms512m -XX:NativeMemoryTracking=summary

I hope the difference between a variable called JAVA_OPTIONS or any env variable and the VM arguments is clear.

Going back to your original question I am not completely sure, but Kubernetes seems to use the JAVA_OPTS environment variable. Check https://dev.to/sandrogiacom/kubernetes-for-java-developers-debug-application-4l1a

With regards to the options for NMT, the official documentation mentions "summary" and "detail".

Ref: https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html

Hope that helps!

`_JAVA_OPTIONS` has special handling in JVM, actually: https://stackoverflow.com/questions/17781405/information-about-java-options — Vadzim, Mar 17 '23 at 16:31

-XX:NativeMemoryTracking not picked up from _JAVA_OPTIONS

1 Answers1