Increase JAVA Heap space for 2gig+ XML File

Question

I have a simple Java program that I want to use to search a large xml file that is about 2 gigs. I am using intellij and I am unable to get it run because I keep an out of memory heapspace error.

I already have intellij compiler set to use the max memory (4 gigs on my machine).

Is there any other way that I can make this work?

When you have a file that big, maybe its a good idea to not keep it whole in memory. — Mephy, Aug 21 '14 at 16:52
Possible duplicate [http://stackoverflow.com/questions/8581501/how-can](http://stackoverflow.com/questions/8581501/how-can-i-give-the-intellij-compiler-more-heap-space) and [http://stackoverflow.com/questions/18631038/setting-heap-](http://stackoverflow.com/questions/18631038/setting-heap-size-in-intellij-idea-correctly) — Darshan Patel, Aug 21 '14 at 16:54
2g? Maybe a common SAX Parser is the better way instead of using a DOM Parser. Do you really need everything in memory? Besides: are you sure having heap size increased for your application? Or is it just heap size for your editor? — atmin, Aug 21 '14 at 16:55

score 2 · Answer 1 · answered Aug 21 '14 at 16:56

2

You probably want to use a stream-based approach, using, say, Stax or Jackson, so that you don't have to hold the whole document in memory at once.

answered Aug 21 '14 at 16:56

GreyBeardedGeek

29,460
2
47
67

score 1 · Answer 2 · answered Aug 21 '14 at 17:15

Depending on the details of the XML file in question, 4GB - or even 8GB - may not be enough if you are using a DOM-style parser. For example, ASCII characters need two bytes when represented within a Java string, and there is quite a bit of overhead when the objects that correspond to the various XML elements are created. Loading the whole file as an object hierarchy in memory generally requires significantly more space than the size of the file.

If you are just searching within an XML document, you probably do not require random access to its elements. In that case, a serial access model such as SAX is better suited. Java already contains a SAX parser, as mentioned in this tutorial from Oracle.

score 0 · Answer 3 · answered Aug 21 '14 at 16:55

0

You can use jvisualvm which is in your jdk/bin folder to monitor and modify heap memory.

answered Aug 21 '14 at 16:55

Mohammad Karimi

77
1
1
10

score 0 · Answer 4 · edited May 06 '15 at 21:31

The “java.lang.OutOfMemoryError: Java heap space” error will be triggered when you try to add more data into the heap space area in memory, but the size of this data is larger than the JVM can accommodate in the Java heap space.

Note that the amount of memory your Java application is eligible to consume is specified during the JVM startup (via -Xmx and -XX:MaxPermSize parameters for example). And in your case, giving more memory to compiler will not help the runtime - try increasing the max allowed heap size of the actual runtime instead of the compiler.

It is generally as easy as adding or increasing the value of parameter similar to the following, where the com.mycompany.MyProgram is allowed to use up to 1G of heap memory:

java -Xmx1024m com.mycompany.MyProgram

Increase JAVA Heap space for 2gig+ XML File

4 Answers4