Out of Memory when reading file in parallel

Question

I am doing test to find out the best way to read and process csv file. So I need to read each line of the csv file and analyse each of them. So basically all works fine for a file containing thousands of rows. However when trying with a CSV file containing more than 1 millions rows I am getting an out of memory exception. I thought Stream Parallel would perform faster. So I am bit confused why I got this out of memory error. How does Java process the parallel reading?

Below is the test code reading file sequentially and in parallel.

String filename = "c:\\devs\\files\\datas.csv"; // 193MB
Path path = Paths.get(filename);

@Test
public void testFileExist() {
    assertTrue(Files.exists(path));
}

@Test
public void testSingleThreadRead() {
    Function<Path, String> processfile = (Path p) -> {
        String result = "";
        try {
            result = Files.lines(p).collect(Collectors.joining(" ,"));
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        return result;
    };

    long start = System.currentTimeMillis();
    String result = processfile.apply(path);
    long end = System.currentTimeMillis();
    assertFalse(result.isEmpty());
    System.out.println(end -start + "ms");
}

@Test
public void testSingleThreadReadParallel() {
    Function<Path, String> processfile = (Path p) -> {
        String result = "";
        try {
            result = Files.lines(p).parallel().collect(Collectors.joining(" ,"));
        } catch (IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }

        return result;
    };

    long start = System.currentTimeMillis();
    String result = processfile.apply(path);
    long end = System.currentTimeMillis();
    assertFalse(result.isEmpty());
    System.out.println(end -start + "ms");
}

Exception

java.lang.OutOfMemoryError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.util.concurrent.ForkJoinTask.getThrowableException(Unknown Source)
at java.util.concurrent.ForkJoinTask.reportException(Unknown Source)
at java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(Unknown Source)
at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.util.stream.ReferencePipeline.collect(Unknown Source)
at test.TestProcessFile.lambda$1(TestProcessFile.java:48)
at test.TestProcessFile.testSingleThreadReadParallel(TestProcessFile.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at 

Update

Running parallel processing in separe class and still got this exception

Exception in thread "main" java.lang.OutOfMemoryError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at java.util.concurrent.ForkJoinTask.getThrowableException(Unknown Source)
at java.util.concurrent.ForkJoinTask.reportException(Unknown Source)
at java.util.concurrent.ForkJoinTask.invoke(Unknown Source)
at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(Unknown Source)
at java.util.stream.AbstractPipeline.evaluate(Unknown Source)
at java.util.stream.ReferencePipeline.collect(Unknown Source)
at ProcessFileParallel.lambda$0(ProcessFileParallel.java:19)
at ProcessFileParallel.main(ProcessFileParallel.java:29)
Caused by: java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuilder.append(Unknown Source)
at java.util.StringJoiner.merge(Unknown Source)
at java.util.stream.Collectors$$Lambda$5/990368553.apply(Unknown Source)
at java.util.stream.ReduceOps$3ReducingSink.combine(Unknown Source)
at java.util.stream.ReduceOps$3ReducingSink.combine(Unknown Source)
at java.util.stream.ReduceOps$ReduceTask.onCompletion(Unknown Source)
at java.util.concurrent.CountedCompleter.tryComplete(Unknown Source)
at java.util.stream.AbstractTask.compute(Unknown Source)
at java.util.concurrent.CountedCompleter.exec(Unknown Source)
at java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(Unknown Source)
at java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

Answer 1

Your code fails at testSingleThreadReadParallel and not parallel. The problem is elsewhere - probably collecting the entire file as a String.

Files.lines are buffered (look at the implementation), so reading the file will most probably not cause any problems.

But collecting that file to a single String will require lots of memory obviously, much more then what the file size itself is.

Actually reading those files in parallel will require much more memory rather than sequential as per my understanding. Each Thread will read it's chuck in memory, in parallel, so your parallel approach will require more memory. And by more I mean your numbers of CPU * BufferSize from Stream.lines.

EDIT2

After taking some time, I realize that your problem has to be somewhere else. Like are do you have actually lines in your files? Or may be you are at the limit - I mean parallel will increase memory indeed, but not with that much. May be you need to increase your -Xms and -Xmx by just a little.

For example I've created for test purposes a file with 247MB of dummy data and ran this code on it:

 Path p = Paths.get("/private/tmp/myfile.txt");
 Stream<String> s = Files.lines(p).parallel(); // and without parallel 
 s.forEach(System.out::println);

The settings that I used are -Xmx200m -Xms200m for both parallel and sequential processing. This is less than the actual file size. Still it works just fine.

Your main problem is that you are collecting everything to a single String, thus making it HUGE in size. Collecting everything to String on my machine under jdk-8 requires at least 1.5GB of Heap.

Also a very good read here

Answer 2

Try to change your JVM's memory settings in the JVM args, especially the -Xmx (minimum heap memory) arg. See Oracle's Documentation.

Another (and even better) option is to read your file in chunks, as suggested in the comments. This will ensure the maximum memory size used for reading the file.