Multiple file readers to read same file at same time in parallell

Question

I have a very large text data file. Is it possible to read this file by multiple file readers from different locations at the same time in parallel? For example one reader starts reading it from start and goes to middle, another starts reading from the middle to the end of the file.

I have an alternate approach of reading file via stream and using parallel. But it does not achieve the goal.

  Files.lines(filePath).parallel

Multiple file readers might not read the same file because the file has already been taken and locked by another thread that is currently reading the file. Is there anyway to make this file shared among many threads and they can concurrently read it.

https://stackoverflow.com/questions/10624899/multithreading-a-massive-file-read — Amit Kumar Lal, Aug 08 '18 at 06:47
Kindly search StackOverflow .. your question may already have an answer here :) — Amit Kumar Lal, Aug 08 '18 at 06:48
@Hades Thanks for sharing above link. It's quite helpful to determine my approach. — malware, Aug 08 '18 at 07:02

score 1 · Answer 1 · answered Aug 08 '18 at 06:44

Is it possible to read this file by multiple file readers from different locations at the same time in parallel?

Yes. You can read the file from multiple points at once using RandomAccessFile. You can only do this if the file isn't locked.

If you have a text file you need to find where the start of the line is and the built-in tools don't really support this class so you will have to do some of the work yourself.

Is there any way to make this file shared among many threads and they can concurrently read it.

We have a couple of tools which support both concurrent reads and writes to the same file across processes so it is possible. (They work on binary files, not text)

Thanks @Peter. As per your suggestion to read the binary files concurrently, can you please name any tool/lib? — malware, Aug 08 '18 at 06:56
We have two tools which use this approach for a different version. — Peter Lawrey, Aug 08 '18 at 07:08

Multiple file readers to read same file at same time in parallell

1 Answers1