0

I'm dealing with a very large file (~250GB) of floats and need to constantly access its data as well as modifying it; there's no good hot spot to cache the data. So I'm considering to use memorymappedfile. However I'm not quite sure how JVM and OS will handle de-mapping and resource releases.
Say I'm mapping regions with chunk size ~100MB from the original 250GB file. Eventually, all 250GB of data will be mapped and if none of the previous mappings gets released from RAM, wouldn't it still crash the JVM or OS for running out of memory? If such mappings are in fact being released, what's the mechanism behind?
Thanks!

Allen Ai
  • 63
  • 1
  • 5
  • No. You should do some **research**, e.g. look at [10 Things to Know about Memory Mapped File in Java](https://www.codeproject.com/Tips/683614/Things-to-Know-about-Memory-Mapped-File-in-Java), to know more. --- But you should also consider just using `RandomAccessFile`, depending on your actual use. The OS will cache that too, so whether the performance is good enough, only testing can tell. – Andreas Aug 03 '18 at 22:57
  • *"wouldn't it still crash the JVM or OS for running out of memory?"* No. --- Based on this in the linked article: *Most of the major operating systems like Windows platform, UNIX, Solaris and other UNIX like operating system supports memory mapped IO and with 64 bit architecture **you can map almost any file into memory** and access it directly using Java programming language.* --- As long as you don't exceed *addressable memory space*, you're good. – Andreas Aug 03 '18 at 23:04
  • @Andreas Correct me if I'm wrong: virtual address space is the only limit in this specific scenario (x86-64)? – Allen Ai Aug 03 '18 at 23:05
  • @Andreas - apart from vm thrashing .... – Stephen C Aug 04 '18 at 03:21

1 Answers1

0

Say I'm mapping regions with chunk size ~100MB from the original 250GB file. Eventually, all 250GB of data will be mapped and if none of the previous mappings gets released from RAM, wouldn't it still crash the JVM or OS for running out of memory?

No. If the initial request to map the file succeeded, then accessing the mapped file won't directly crash the JVM or the OS1.

If such mappings are in fact being released, what's the mechanism behind?

Short answer: the OS's virtual memory system.

Note that the mappings that you are actually talking about are the mappings between physical RAM pages and virtual addresses. The virtual memory system on a typical OS handles this transparently. If it runs out of physical RAM pages, it flushes dirty memory pages to disk and reassigns them to other virtual addresses.

References:

(There are lots of other StackOverflow Q&A's on the topic of virtual memory, etcetera.)

1 - Actually, it might ... if you push too hard. Suppose that you map a file that is substantially bigger than the amount of physical RAM. Then suppose that you aggressively read / write it in a random pattern. This could generate so much virtual memory I/O traffic, that "thrashing" causes other applications and the OS itself become unusable. On Linux, this is liable to cause the "OOM killer" process to kill the JVM.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216