How to get list of file in directory in batches?

Question

I have a location where 3000 files is stored. But i want to get the list of 1000 files at a time and in next call another 1000 files and so on.

Please find my below code :

import java.io.File;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.Comparator;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class FileSystem {

    public static void main(String args[]) throws Exception {
        FileSystem.createListFile();
        FileSystem.getFileInBatch();
    }

    private static void getFileInBatch() {
        int MAX_INDEX = 1000;
        List<String> result = new ArrayList<>();

        try (Stream<Path> walk = Files.walk(Paths.get("C://FileTest"))) {

            List<String> batchResult = walk.filter(p -> Files.isRegularFile(p) && p.getFileName().toString().endsWith(".txt"))
                    .sorted(Comparator.comparingInt(FileSystem::pathToInt))
                    .map(x -> x.toString()).limit(MAX_INDEX).collect(Collectors.toList());

            batchResult.forEach(System.out::println);
            System.out.println(batchResult.size());
            result.addAll(batchResult);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }



    private static int pathToInt(final Path path) {
        return Integer.parseInt(path.getFileName()
                .toString()
                .replaceAll("Aamir(\\d+).txt", "$1")
        );
    }

    private static void createListFile() throws IOException {
        for (int i = 0; i < 3000; i++) {
            File file = new File("C://FileTest/Aamir" + i + ".txt");

            if (file.createNewFile()) {

                System.out.println(file.getName() + " is created!");
            }
        }
    }

}

I am able to get the first 1000 (Aamir0.txt to Aamir999.txt) files using the limit in streams.

Now how can i get the next 1000 files ( Aamir1000.txt to Aamir1999.txt) in batches and then the next 1000 files. Im expecting some loops who will continuously give 1000 ordered files in batches.

Please, take a look onto https://stackoverflow.com/questions/30641383/java-8-stream-with-batch-processing it's may help — RoninDev, May 07 '20 at 08:03
No, it will be a huge number of files. i was trying for first do it 3000 files so we can do it same — codehunter, May 07 '20 at 08:40
@AAMIRAZAM Okay. Also, what exactly are you putting into the list? The path to all the files? I'm having a hard time understanding your code. — cegredev, May 07 '20 at 08:43
@Schred: There is a location where huge number of files are stored let say 4000 files. Now I want to get the pick 1000 files at a time in a batch and do some operation and then pick up the next 1000 files and do some operation and next pick the next 1000 files and do the operation. This will continue until I won't visit all the files in the location. — codehunter, May 07 '20 at 12:35
@ Schred: In the list, it gets the filename of the 1000 files — codehunter, May 07 '20 at 12:36

cegredev · Accepted Answer · 2020-05-07T14:34:58.300

1

I'm not sure what you want to do with each file, but this logic should work for you:

private static final int MAX_VALUE = 5;

public static void main(String[] args) {
    try (Stream<Path> walk = Files.walk(Paths.get("path/to/your/directory"))) {

        List<String> list = new ArrayList<>(MAX_VALUE);
        walk.filter(path -> path.getFileName().toString().endsWith("txt"))
            .sorted((a, b) -> compare(a, b)
            .forEach(path -> handlePath(path, list));

    } catch (IOException e) {
        e.printStackTrace();
    }
}

public static void handlePath(Path path, List<String> list) {
    list.add(path.toString());

    if (list.size() >= MAX_VALUE) {
        // Do something

        list.clear();
    }
}

public static int compare(Path a, Path b) {
    String aStr = a.getFileName().toString(), bStr = b.getFileName().toString();

    int valA = Integer.parseInt(aStr.replaceAll("[^0-9.]|\\.", ""));
    int valB = Integer.parseInt(bStr.replaceAll("[^0-9.]|\\.", ""));

    return valA - valB;
}

edited May 07 '20 at 14:34

answered May 07 '20 at 08:47

cegredev

1,485
2
11
25

It's nice but in this case, I should know the filename but suppose we don't know the filename . then does it worked? – codehunter May 07 '20 at 12:49
Suppose there are file with different names such abc.txt, bred.txt, hfc.txt. Then does it worked – codehunter May 07 '20 at 12:50
1

@AAMIRAZAM Ah, okay. Didn't think that would be a requirement. You do have the directory though, don't you? – cegredev May 07 '20 at 12:52
Yeah, I have a directory : C://filetest but how it will get sorted and pick the 1000 files in a batch from directory – codehunter May 07 '20 at 12:58
1

@AAMIRAZAM Okay, I have completly edited my answer. This should work. I'm using Java 12 though, I don't know if you can use all of those functions. If not, I'll make some small adjustments. – cegredev May 07 '20 at 13:08
Not working, handle path method is not getting called – codehunter May 07 '20 at 13:15
Superb @Schred . Thank you so much . You are my Saviour. :) – codehunter May 07 '20 at 14:32
How does the compare method works for batch size? Could you explain the logic? @cegredev – Fay007 Feb 07 '23 at 08:47

score 0 · Answer 2 · answered May 07 '20 at 08:08

0

You are using streams and they have fine APIs to do your desire.

you are limiting the reading data with

.limit(MAX_INDEX)

for getting other results one way is to use combination of skip and limit methods:

.skip(MAX_INDEX).limit(MAX_INDEX)

this way you can access the second 1000 characters based on your MAX_INDEX value.

answered May 07 '20 at 08:08

Majid Roustaei

1,556
1
20
39

this won't be optimal since it has to pass through all the first 1000 items all over again – Kavin Eswaramoorthy May 07 '20 at 08:27
Yeah. Also , again i have to write the same code for the next 1000 elements and it willbe repeated. Any one has any idea , how we can do it . – codehunter May 07 '20 at 08:42

How to get list of file in directory in batches?

2 Answers2