Using Hard Disk Space as Java heap?

Question

I am trying to run a jar file which gives 'out of memory error' after running some time. I have searched sometimes about using Hard Disk space as java heap, but most of the solutions suggest optimizing code or using something like 'Ehcache'. But the problem is, I have only the jar file and I did not write its code. So, my question is Can I use some mechanism, or software, by which I can make java use the hard disk space as heap, without modifying the code of the jar file in any way?

Answer 1

I am assuming that you have a JAR file of Java classes without accompanying source code. This is of course problematic, but it's important for you to know what those classes are doing because if there is some insurmountable memory leak that those classes cause (Yes, Java does have memory leaks in the form of the so-called Unintended Object Retentions) then following efforts may still fall short.

Having said that, it should be noted that the JVM (In particular, the Oracle JVM) provides some configurability in terms of how much heap memory you want it to use to run your Java app. The single most important configuration parameter is -Xmx, the so-called "Maximum Heap Memory". When you specify this parameter, like e.g. -Xmx6g, the JVM, at the startup, will ask the Operating System (on which the JVM runs) for that much of memory. The exact details are a bit complicated, but if that amount of memory is not available at the startup, the JVM startup fails. There is complicated algorithm to determine the default value for this heap memory but if you have requirements, you should override the default like above. In the above case, your JVM (java -Xmx6g -jar test.jar) is asking for 6 GB of heap to start with.

Determining the correct value of the heap memory that the JVM would need over its lifetime is somewhat of an art, although there are several things you can consider. The most important thing is understanding the runtime characteristics of your app. You should know how and when the objects are allocated and deallocated (so that the Garbage Collector kicks in) so that you do not run out of the heap space when the next allocation request is made by the JVM to the OS. And it seems to me that this is what is happening in your case -- somewhere, a bunch of objects are being constructed, requiring memory that the OS does not have.

Obviously, the OS will try to see if that much of physical RAM is available and if it does, the JVM's request will be granted. This is the happy path.

This happy path, however, is not always possible in that enough RAM is not available. Even then, the OS can grant the request because of this OS concept called the Virtual Memory. Without going into too many details (you can read up on virtual memory), it would suffice to say that OS has a mechanism where it can use the hard disk space as RAM via the mechanism of page swapping. In other words, it can swap the data into physical RAM or swap data out to hard disk in a way that is transparent to all the processes running on it (including the JVM, which is nothing but a process named java).

This is where your sysadmin skills help you. There are ways to increase the swap space on your OS. For instance, on Linux, you can use the swapon command to do that. This will invariably result in a phenomenon called thrashing, but that's the best you can do (you can also get rid of other programs to save memory).

Again, this might get you only this far. If there's a memory leak in your program, no amount of GC (reclaiming the unused memory, a great trait of the JVM) or swap space may help you and sooner or later you'll get the dreaded OutOfMemoryError.

You should consider specifying -XX:HeapDumpOnOutOfMemoryError to see what the JVM was doing at the time of this error.

Answer 2

You can't use harddisk space for running the java program,

You can do two things :

1. Increase heap memory using the following command

   java -Xmx1024m -jar Test.jar

2. Use Java8 which uses the metaspace which uses the native memory https://dzone.com/articles/java-8-permgen-metaspace