Split the stream

Question

I have a file. I get Stream using Files.lines. The file is big. I need to go through it in a loop and form several arrays. During the passage through the file, each generated array must be passed to the method that will process it. I know that there are PartitioningBy and GroupingBy methods, but I do not know how to apply them to my task. I'm trying to do this:

@Test
public void myTest() {
    Stream<String> lines = Stream.of(
            "some row from my file 1",
            "some row from my file 2",
            "some row from my file 3",
            "some row from my file 4",
            "some row from my file 5",
            "some row from my file n",
            "some row from my file 750000"
    );
    lines.parallel()
            .unordered()
            .collect(Collectors.partitioningBy(s -> s == 3).supplier(it -> {
                myParser(it);
            }));
}

public void myParser(List<String> myList){
    //this piece of code should give the length of the transmitted array
    System.out.println(myList.size()); 
}

In the myParser method, I want to get arrays of 3 elements and process them

@Ruslan it seems he doesn't want an output, he just wants to split the stream in slices of 3 rows. Seems reasonable to you? — LppEdd, Mar 27 '19 at 10:39
@Naman I know that there is a string and I incorrectly compare it. Can I get the length of the intended array there? — maksim2112, Mar 27 '19 at 10:49
Possible duplicate of [Java 8 Stream with batch processing](https://stackoverflow.com/questions/30641383/java-8-stream-with-batch-processing) — Joe, Jun 24 '19 at 13:28

score 0 · Answer 1 · answered Mar 27 '19 at 11:40

I will dwell on this option

    ArrayList<String> list = new ArrayList<>();

    lines.forEach(it -> {
        list.add(it);
        if (list.size() > 0 && list.size() % 3 == 0) {
            myParser(list);
            list.clear();
        }
    });

score 0 · Answer 2 · answered Mar 27 '19 at 11:45

You could try this to split your stream:

public class T30SplitStream {

public static void main(String[] args) {
    Stream<String> lines = Stream.of("some row from my file 1", "some row from my file 2",
            "some row from my file 3", "some row from my file 4", "some row from my file 5",
            "some row from my file n", "some row from my file 750000");
    AtomicInteger i = new AtomicInteger(0);
    Map<Integer, List<String>> map = lines.parallel().unordered().map(s -> new Pair(i.incrementAndGet(), s))
            .collect(Collectors.groupingBy(p -> p.i % 3, Collectors.mapping(p -> p.s, Collectors.toList())));
    System.out.println(map);

}

public static class Pair {
    public final Integer i;
    public final String s;

    public Pair(int i, String s) {
        this.i = i;
        this.s = s;
    }
}
}

score 0 · Accepted Answer · answered Feb 25 '20 at 07:28

Solution:

List<List<String>> partition = ListUtils.partition(lines, 3);
partition.parallelStream().forEach(this::myParser);

maven:

    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-collections4</artifactId>
        <version>4.1</version>
    </dependency>

Split the stream

3 Answers3