Scanner.findInLine() leaks memory massively

Question

I'm running a simple scanner to parse a string, however I've discovered that if called often enough I get OutOfMemory errors. This code is called as part of the constructor of an object that is built repeatedly for an array of strings :

Edit: Here's the constructor for more infos; not much more happening outside of the try-catch regarding the Scanner

   public Header(String headerText) {
        char[] charArr;
        charArr = headerText.toCharArray();
        // Check that all characters are printable characters
        if (charArr.length > 0 && !commonMethods.isPrint(charArr)) {
            throw new IllegalArgumentException(headerText);
        }
        // Check for header suffix
        Scanner sc = new Scanner(headerText);
        MatchResult res;
        try {
            sc.findInLine("(\\D*[a-zA-Z]+)(\\d*)(\\D*)");
            res = sc.match();
        } finally {
            sc.close();
        }

        if (res.group(1) == null || res.group(1).isEmpty()) {
            throw new IllegalArgumentException("Missing header keyword found");     // Empty header to store
        } else {
            mnemonic = res.group(1).toLowerCase();                            // Store header
        }
        if (res.group(2) == null || res.group(2).isEmpty()) {
            suffix = -1;
        } else {
            try {
                suffix = Integer.parseInt(res.group(2));       // Store suffix if it exists
            }  catch (NumberFormatException e) {
                throw new NumberFormatException(headerText);
            }
        }
        if (res.group(3) == null || res.group(3).isEmpty()) {
            isQuery= false;
        } else {
            if (res.group(3).equals("?")) {
                isQuery = true;
            } else {
                throw new IllegalArgumentException(headerText);
            }
        }

        // If command was of the form *ABC, reject suffixes and prefixes
        if (mnemonic.contains("*") 
                && suffix != -1) {
            throw new IllegalArgumentException(headerText);
        }
    }

A profiler memory snapshot shows the read(Char) method of Scanner.findInLine() to be allocated massive amounts of memory during operation as a I scan through a few hundred thousands strings; after a few seconds it already is allocated over 38MB.

I would think that calling close() on the scanner after using it in the constructor would flag the old object to be cleared by the GC, but somehow it remains and the read method accumulates gigabytes of data before filling the heap.

Can anybody point me in the right direction?

Answer 1

You haven't posted all your code, but given that you are scanning for the same regex repeatedly, it would be much more efficient to compile a static Pattern beforehand and use this for the scanner's find:

static Pattern p = Pattern.compile("(\\D*[a-zA-Z]+)(\\d*)(\\D*)");

and in the constructor:

sc.findInLine(p);

This may or may not be the source of the OOM issue, but it will definitely make your parsing a bit faster.

Related: java.util.regex - importance of Pattern.compile()?

Update: after you posted more of your code, I see some other issues. If you're calling this constructor repeatedly, it means you are probably tokenizing or breaking up the input beforehand. Why create a new Scanner to parse each line? They are expensive; you should be using the same Scanner to parse the entire file, if possible. Using one Scanner with a precompiled Pattern will be much faster than what you are doing now, which is creating a new Scanner and a new Pattern for each line you are parsing.

Answer 2

The strings that are filling up your memory were created in findInLine(). Therefore, the repeated Pattern creation is not the problem.

Without knowing what the rest of the code does, my guess would be that one of the groups you get out of the matcher is being kept in a field of your object. Then that string would have been allocated in findInLine(), as you see here, but the fact that it is being retained would be due to your code.

Edit:

Here's your problem:

mnemonic = res.group(1).toLowerCase();

What you might not realize is that toLowerCase() returns this if there are no uppercase letters in the string. Also, group(int) returns a substring(), which creates a new string backed by the same char[] as the full string. So, mnemonic actually contains the char[] for the entire line.

The fix would just be:

mnemonic = new String(res.group(1).toLowerCase());

Answer 3

I think that your code snippet is not full. I believe you are calling scanner.findInLine() in loop. Anyway, try to call scanner.reset(). I hope this will solve your problem.

Answer 4

The JVM apparently does not have time to Garbage collect. Possibly because it's using the same code (the constructor) repeatedly to create multiple instances of the same class. The JVM may not do anything about GC until something changes on the run time stack -- and in this case that's not happening. I've been warned in the past about doing "too much" in a constructor as some of the memory management behaviors are not quite the same when other methods are being called.

Answer 5

Your problem is that you are scanning through a couple hundred thousand strings and you are passing the pattern in as a string, so you have a new pattern object for every single iteration of the loop. You can pull the pattern out of the loop, like so:

    Pattern toMatch = Pattern.compile("(\\D*[a-zA-Z]+)(\\d*)(\\D*)")

    Scanner sc = new Scanner(headerText);
    MatchResult res;

    try {
        sc.findInLine(toMatch);
        res = sc.match();
    } finally {
        sc.close();
    }

Then you will only be passing the object reference to toMatch instead of having the overhead of creating a new pattern object for every attempt at a match. This will fix your leak.

Answer 6

Well I've found the source of the problem, it wasn't Scanner exactly but the list holding the objects doing the scanning in the constructor.

The problem had to do with the overrun of a list that was holding references to the object containing the parsing, essentially more strings were received per unit of time than could be processed and the list grew and grew until there were no more RAM. Bounding this list to a maximum size now prevents the parser from overloading the memory; I'll be adding some synchronization between the parser and the data source to avoid this overrun in the future.

Thank you all for your suggestions, I've already made some changes performance wise regarding the scanner and thank you to @RobI for pointing me to jvisualvm which allowed me to trace back the exact culprits holding the references. The memory dump wasn't showing the reference linking.