Parallel stream creates only one thread and gives result as fast as normal stream

Question

Below is the code where I try to process lines read by from the file in parallel stream and in Normal stream. Surprisingly, parallel stream gives no improvements over normal stream . Am I missing something here ?

Files.walk(Paths.get(tweetFilePath + LocalDate.now())).forEach(
            filePath -> {
                if (Files.isRegularFile(filePath) && !filePath.toString().endsWith(".DS_Store")) {
                    long startTime = System.currentTimeMillis();
                    try {

                        Files.lines(filePath).parallel().forEach(line -> {
                                try {
                                    System.out.println(line);

                                } catch (Exception e) {
                                    System.out.println("Not able to crunch"+ e);
                                }

                        });
                    } catch (Exception e) {
                        System.out.println("Bad line in file ");
                    }finally {
                        System.out.println("total time required:" + (System.currentTimeMillis() - startTime));

                    }   
                }
            });

In this situation, the bottleneck is the I/O source - if you have one physical hard drive then you can parallelise however much you want, it won't go faster... — assylias, Jun 12 '15 at 22:55
maybe instead of parallelizing on lines you could parallelize on files — the8472, Jun 13 '15 at 05:32

Aishwar · Accepted Answer · 2015-06-12T22:46:55.477

Looks like currently, Files.lines reads the file linearly, so the parallel call cannot split the source stream into sub-streams for parallel processing.

See here for details. Relevant section quoted below:

What if my source is based on IO?

Currently, JDK IO-based Stream sources (for example BufferedReader.lines()) are mainly geared for sequential use, processing elements one-by-one as they arrive. Opportunities exist for supporting highly efficient bulk processing of buffered IO, but these currently require custom development of Stream sources, Spliterators, and/or Collectors. Some common forms may be supported in future JDK releases.

That appears to be correct. I collected all the lines from the file in to a list and then parallel streamed on it. It's significantly fast now. Thanks! — Sagar, Jun 13 '15 at 14:52

score 1 · Answer 2 · edited May 23 '17 at 11:46

The first problem is that Files.lines parallelize badly especially for files shorter than 1024 lines. Check this question for details. If you know in advance that your file is short enough to fit in memory, it would be better to read it sequentially to the List first:

Files.readAllLines(filePath, StandardCharsets.UTF_8).parallelStream()...

I have some ideas on how to improve this, but it's still not ideal solution. The fact is the Stream API parallelization is quite ineffective if you cannot even estimate the elements count in the input stream.

The second problem is your forEach operation. Here you just use System.out, so all the threads will try to write to the same PrintStream fighting for the same resource, thus most of the time will be spent in waiting for the lock release. Internally it uses BufferedWriter where all writes are synchronized. You may benefit from parallelization if you don't use shared resources in parallel operations.

By the way Files.lines creates a stream over BufferedReader. It's better to manage it with try-with-resources statement. Otherwise the files will be closed only when underlying FileInputStream objects are garbage-collected, so you may sporadically have errors like "too many open files".

Parallel stream creates only one thread and gives result as fast as normal stream

2 Answers2