String.split() temporary objects and Garbage Collect

Question

In my project, we have a requirement to read a very large file, where each line has identifiers separated by a special character ( "|"). Unfortunately I can't use parallelism, since it is necessary to make a validation between the last character of a line with the first of the next line, to decide whether it will or not be extracted. Anyway, the requirement is very simple: break the line into tokens, analyze them and store only some of them in memory. The code is very simple, something like below:

final LineIterator iterator = FileUtils.lineIterator(file)
while(iterator.hasNext()){
   final String[] tokens = iterator.nextLine().split("\\|");
   //process
}

But this little piece of code is very, very inefficient. The method split() generates too many temporary objects that are not been collected (as best explained here: http://chrononsystems.com/blog/hidden-evils-of-javas-stringsplit-and-stringr.

For comparison purposes: a 5mb file was using around 35 mb memory at the end of file process.

I tested some alternatives like:

• Using a pre compiled pattern (Performance of StringTokenizer class vs. split method in Java)
• Use Guava's Splitter (Java split String performances)
• Optimize String storage (http://java-performance.info/string-packing-converting-characters-to-bytes/)
• Use of optimized collections (http://blog.takipi.com/5-coding-hacks-to-reduce-gc-overhead)

But none of them appears to be efficient enough. Using JProfiler, I could see that the amount of memory used by temporary objects is too high (35 mb used, but only 15 mb is actually been used by valid objects).

Then I decide make a simple test: after 50,000 lines read, explicit call to System.gc(). And then, at the end of process, the memory usage has decreased from 35 mb to 16mb. I tested many, many times, and always got the same result.

I know invoke that invoke of System.gc () is a bad practice (as indicated in Why is it bad practice to call System.gc()?). But is there is any other alternative in a cenario where the split() method could be invoked millions of times?

[UPDATE] I use a 5 mb file only for test purpose, but the system should process much larger files (500Mb ~ 1Gb)

Answer 1

The first and most important thing to say here is, don't worry about it. The JVM is consuming 35MB of RAM because it's configuration says that's a low enough amount. When its highly efficient GC algorithm decides it's time, it will sweep all those objects away, no problem.

If you really want to, you can invoke Java with memory management options (e.g. java -Xmxn=...) -- I suggest it's not worth doing unless you're running on very limited hardware.

However, if you really want to avoid allocating an array of String each time you process a line, there are many ways to do so.

One way is to use a StringTokenizer:

    StringTokenizer st = new StringTokenizer(line,"|");

    while (st.hasMoreElements()) {
        process(st.nextElement());
    }

You could also avoid consuming a line at a time. Get your file as a stream, use a StreamTokenizer, and consume one token at a time in this way.

Read the API docs for Scanner, BufferedInputStream, Reader -- there are lots of choices in this area, because you're doing something fundamental.

However, none of these will cause Java to GC sooner or more aggressively. If the JRE doesn't consider itself short of memory, it won't collect any garbage.

Try writing something like this:

public static void main(String[] args) {
    Random r = new Random();
    Integer x;
    while(true) {
        x = Integer.valueof(r.nextInt());
    }
}

Run it and watch your JVM's heap size as it runs (put a sleep in if the usage shoots up too quickly to see). Each time around the loop, Java creates what you call a 'temporary object' of type Integer. All of these stay in the heap until the GC decides it needs to clear them away. You'll see that it won't do this until it reaches a certain level. But when it reaches that level, it will do a good job of ensuring that its limits are never exceeded.

Answer 2

You should adjust your way of analyzing situations. While the article about the regex compilation under the hood is correct in general, it doesn’t apply here. When you look at the source code of String.split(String), you’ll see that it just delegates to String.split(String,int) which has a special code path for patterns consisting of just one literal character, including escaped ones like your \|.

The only temporary object created within that code path is an ArrayList. The regex package is not involved at all; this fact might help you understanding why precompiling a regex pattern did not improve the performance here.

When you use a Profiler to come to the conclusion that there are too many objects, you should use it also to find out what kinds of objects there are and where they originate, instead of doing wild guessing.

But it’s not clear, why you complain at all. You can configure the JVM to use a certain maximum memory. As long as that maximum has not been reached, the JVM just does what you told it, using that memory rather than wasting CPU cycles just to not using the available memory. Where’s the sense in not using the available memory?