3

I want to covert all files inside ROOT directory into java 8 stream of InputStream.

suppose there are a lot of files inside the ROOT

for example file OS:

ROOT
 |--> file1
 |--> file2

I want to do somethink but using java 8 with the best PERFORMANCE!!

List<InputStream> ins = new ArrayList<>();
File[] arr =  new File(ROOT).listFiles()
for(File file: arr){
  ins.add(FileUtils.openInputStream(file));
}
Flown
  • 11,480
  • 3
  • 45
  • 62
VitalyT
  • 1,671
  • 3
  • 21
  • 49
  • 1
    What exactly is the problem you're trying to solve? Do you want to read multiple files concurrently? Why do you want to have multiples `InputStream`? – Tunaki May 02 '16 at 11:22
  • I want to make copy&paste all files inside my local disk directory into google cloud storage. As i know there is no way to make copy all directory , so i need to do that file by file – VitalyT May 02 '16 at 11:24
  • 1
    Ah so you want to copy files recursively? Like here? http://stackoverflow.com/questions/29076439/java-8-copy-directory-recursively – Tunaki May 02 '16 at 11:25
  • recursive is very slow , i have flat hierarchy...do u have any idea of uploading folder with files into cloud ? – VitalyT May 02 '16 at 11:29
  • 1
    Did you measure it? And you hierarchy isn't flat, there are multiple sub directories. – Tunaki May 02 '16 at 11:29
  • I know i'll edit my question into flat ( without sub-dir) – VitalyT May 02 '16 at 11:33
  • 1
    Something like this then http://stackoverflow.com/questions/26213409/list-of-files-in-directory-with-new-file-api – Tunaki May 02 '16 at 11:35
  • finally the solution : List ins = Files.list(Paths.get("/Users/vitaly")).sorted().map(l -> { try { return Files.newInputStream(l); } catch (IOException e) { e.printStackTrace(); } return null; }).collect(Collectors.toList()); – VitalyT May 02 '16 at 11:52
  • @VitalyTarasiuk, it's ok to post answers to your own questions. As a comment, your answer is hidden under "more comments". If you post as an answer we can review your own solution and offer suggestions for improvement. – Hank D May 02 '16 at 13:45

2 Answers2

1

Short answer: best solution is the one that you have dismissed:

  • Use File.listFiles() (or equivalent) to iterate over the files in each directory.
  • Use recursion for nested directories.

Lets start with the performance issue. When you are uploading a large number of individual files to cloud storage, the performance bottleneck is likely to be the network and the remote server:

  • Unless you have an extraordinarily good network link, a single TCP stream won't transfer data anywhere like as fast as it can be read from disk (or written at the other end).

  • Each time you transfer a file, there is likely to be an overhead for starting the new file. The remote server has to create the new file, which entails adding a directory entry, and inode to hold the metadata, etc.

  • Even on the sending side, the OS and disc overheads of reading directories and metadata are likely to dominate the Java overheads.

(But don't just trust what I say ... measure it!)

The chances are that the above overheads will be orders of magnitude greater than you can get by tweaking the Java-side file traversal.

But ignoring the above, I don't think that using the Java 8 Stream paradigm would help anyway. AFAIK, there are no special high performance "adapters" for applying streams to directory entries, so you would most likely end up with a Stream wrapper for the result of listFiles() calls. And that would not improve performance.

(You might get some benefit from parallel streams, but I don't think you will get enough control over the parallelism.)

Furthermore, you would need to deal with the fact that if your Java 8 Stream produces InputStream or similar handles, then you need to make sure that those handles are properly closed. You can't just close them all at the end, or rely on the GC to finalize them. If you do either of those, you risk running out of file descriptors.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
0

InputStream is = new ByteArrayInputStream(Files.readAllBytes(Paths.get(Root)));

suulisin
  • 1,414
  • 1
  • 10
  • 17
  • to your suggestion : java.io.IOException: Is a directory at sun.nio.ch.FileDispatcherImpl.read0(Native Method) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) at sun.nio.ch.IOUtil.read(IOUtil.java:197) at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:159) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:65) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:109) at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.nio.file.Files.read(Files.java:3105) at java.nio.file.Files.readAllBytes(Files.java:3158) – VitalyT May 02 '16 at 11:32
  • sorry i did not test it at my end but look in that line – suulisin May 02 '16 at 11:53