3

I need to read the contents of a binary file containing floats that represent the positions of a big number of particles, and pass this information to my program so they can be rendered to the screen. The file contains the position of every particle at every frame, and thus can be as big as 10 or more GB.

My plan is to fill all available memory with the next frames the program will need. Once a frame has been shown, I need to free that space and load a new frame from the file. If a certain frame is requested (say, frame 12), I need to go to this frame in the file and read it, as well as all other successive frames that fit in the memory.

The question is, how can I read, and then store this information in a way that is efficient?

I have done some research, and read similar questions. Arrays and Vectors are not an option since they can fit about 2gb of data, when I could be loading more than that to ram. Besides, initializing an array of such a size seems to lead to an "out of memory exception", and expanding an array takes too much time.

Buffer a large file; BufferedInputStream limited to 2gb; Arrays limited to 2^31 bytes

MappedByteBuffer could be an option, but it is also limited to 2 gb, and I do not understand exactly how it works.

I could also use external libraries that save data outside of the buffer (but which one?), or use a ByteBuffer, I guess, but what is the best option in this case?

In short, I need to read the file fast (probably in parts, unless there is another way), and save as much in memory as I can, and pass this information to another thread, frame by frame. How do I read this file, and where do I store the data in memory?

Read large files in Java This source seems to address something similar, but it is about reading text, and it seems he could use a MappedByteBuffer or a BufferedInputStream, since his file is of only 1.5 GB.

  • 2
    If reading binary files, you should have a look at the `java.nio.*` package. It suits what you're trying to achieve. Still, the best way is for you to test options and figure out the best for your case. – Magd Kudama Apr 24 '18 at 20:10
  • 2
    See javadoc of packages [`java.nio`](https://docs.oracle.com/javase/8/docs/api/java/nio/package-summary.html#package.description) and [`java.nio.channels`](https://docs.oracle.com/javase/8/docs/api/java/nio/channels/package-summary.html#package.description) – Andreas Apr 24 '18 at 20:20
  • But aren't all the buffers in the nio package limited to a size of 2 GB? –  Apr 24 '18 at 21:01
  • 1
    I fail to see the limitation of the 2GB maximum on buffer sizes. Each frame is going to be less than 1GB (based on your file size of 10GB or more and requesting the 12th frame). So allocate `N` buffers (such as `FloatBuffer` or `DoubleBuffer`), each large enough to hold exactly one frame, and read successive frames into each buffer. When you’re finished displaying frame `i`, reuse the buffer to read the `i+N` frame from the file, while displaying frame `i+1`. You don’t need (or want) to free the buffers, just maintain your own “free list” to reuse them, thus avoiding garbage collection. – AJNeufeld Apr 25 '18 at 00:31
  • You could look at netty(https://netty.io/) which is quite scalable. – nitnamby Apr 28 '18 at 07:50

0 Answers0