2

I have a Java project which shows a memory leak in the product environment. We can see its memory expand and grow bigger and bigger using 'top' or 'cat /proc/PID/status|grep VmRSS', but its JVM heap remains in a good state, which means the heap memory is in control by '-Xmx' and other JVM options. I had used jmap/dump/jstack and any other command I know to analyze what's wrong, but in vain.

Finally, we found the problem by testing line by line. It's a problem in the JVM JavaScript engine. I extracted the code here which can reproduce the problem:

package com.unionpay.cqp.arch.js;


import javax.script.*;


import static java.lang.Thread.sleep;


public class JsEngineMain {
    private static final ScriptEngine SCRIPT_ENGINE = new ScriptEngineManager().getEngineByName("JavaScript");
    private static final String SCRIPT_FUNC_1 = "function max_num(a, b){return (a>b)?a:b;} ";


    public static void main(String[] args) {
        if (args == null || args.length < 1) {
            System.out.println("wrong args, use like:");
            System.out.println("java -jar xxx.jar SLEEP_TIME_MILLIES");
            System.out.println("java -cp xxx.jar com.unionpay.cqp.arch.js.JsEngineMainBak SLEEP_TIME_MILLIES");
            return;
        }
        long sleepTime = Long.parseLong(args[0]);
        try {
            while (true) {
                sleep(sleepTime);
                testLoopFuncString();
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
    }


    private static void testLoopFuncString() {
        try {
            Compilable compilable = (Compilable) SCRIPT_ENGINE;
            CompiledScript compiledScript = compilable.compile(SCRIPT_FUNC_1);
            compiledScript.eval();
            Invocable invocable = (Invocable) SCRIPT_ENGINE;
            Object res = invocable.invokeFunction("max_num", 6, 1);
            showRes(res);
        } catch (Exception e) {
            e.printStackTrace();
        }
    }


    private static void showRes(Object res) {
        if (System.currentTimeMillis() % 20000 == 1) {
            System.out.println(res);
        }
    }
}

You can run it with the command:

java -Xmx 128M -XX:MaxMetaspaceSize=64M -XX:MaxDirectMemorySize=64M -cp js-engine-test.jar com.unionpay.cqp.arch.js.JsEngineMain 2

and monitor the memory usage with:

while true; do cat /proc/PID/status|grep VmRSS;sleep 5;done

Then you will see the memory usage grow over time.

Well, my question is, how to find the problem if we don't know where the problem is? And how do we find the incorrect code by memory analysis or command tools?

I use pmap and jcmd to see where the memory is occupied, I can get some info from jcmd tools and I know it's a off-heap problem. but I can not connect it with codes, so I can not find which line of my code is incorrect.

sheen
  • 21
  • 3
  • This sounds like an issue that should be profiled. Try [Visual VM](https://visualvm.github.io/) to get a general picture of the memory consumption, and then try to profile the memory of the application to see where the memory is being used. – filpa Nov 02 '20 at 09:13
  • 1
    When I run the command as you described, I get an exception, due to the line `String cmd = args[1];` (that apparently serves no purpose, except printing a cryptic message). Besides that, you should not mix `System.err` and `System.out` in what is supposed to be a single message. Regarding your described problem, memory growth is not wrong, as long as you’ve told the JVM that it is ok to use that memory and there’s no actual failure due to insufficient memory. Set a tight maximum heap memory and check whether it fails when reaching that limit or just performs garbage collection as expected… – Holger Nov 02 '20 at 11:51
  • I'm sorry that the code is simplified from my running code because my origin source code has some unrelated codes. I have updated the code above. I will do a test as you say, but I did not mean it's something wrong with jvm, I just want to find where the problem is or which place casues the memory grow bigger and bigger. – sheen Nov 03 '20 at 00:56
  • 1
    It looks to be a JVM issue anyway, perhaps related to Nashorn Engine. The following bug seems similar https://bugs.openjdk.java.net/browse/JDK-8197544 , which is a duplicated of JDK-8081323 – devwebcl Nov 03 '20 at 02:04

0 Answers0