1

I have a 1.5G xml file, and I use a DOM java parser (after throw out this problem I know DOM is not a good tool for big data, while I am still curious about the problem below). My issue is "OutOfMemoryError: Java heap space", based on the answers already exist, I change the eclipse.ini -xms and -xmx size both to 8096m, and I show the heap status in eclispe window to monitor how much heap size has used. While when I run this code, it just used "80m/8096m" then throw out the "out memory" bug, I wonder why there is clearly huge space not used, i.e. "8096m - 80m ", but still get to out of memory.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
michael
  • 39
  • 7
  • 3
    "Bug"? No, your code is doing it. Try profiling the code with something like Visual VM to see where memory is being consumed. – duffymo Jul 17 '16 at 01:24
  • 3
    If it is your program that is using all the memory and not eclipse, you need to change the VM args that eclipse uses to run *your program*, not the VM args for ecliipse itself. See http://stackoverflow.com/questions/4175188/setting-memory-of-java-programs-that-runs-from-eclipse for how to do this – wakjah Jul 17 '16 at 01:28
  • For 1.5 GB xml and -Xmx8096m if your code is still throwing OOME then I am almost certain there is something that in the code that needs improvement. Usually for huge data types SAX parsing is better. Also how are you specifying Xmx ? can you show us ? – SomeDude Jul 17 '16 at 01:29
  • 2
    You need to modify the JVM that runs your program, not the one that runs Eclipse – OneCricketeer Jul 17 '16 at 01:40
  • 1
    A 1.5 GB xml file, you say? A SAX parser would probably be a better choice. DOM parsers have to load all of the data and metadata into the memory. That can be memory intensive. SAX, on the other hand, is an event driven parser. Thus it only has to load a bit of that xml file into memory at a time. If there isn't a specigic reason for you to load the entire file into memory I would avoid it... – CodeJockNYC Jul 17 '16 at 01:47
  • Thanks for all you guys quick feedback. Now I may understand jvm and eclipse memory are different things, and SAX is better in my case. – michael Jul 17 '16 at 02:27
  • Another interesting question may be my xml size is 1.5G, could I estimate the memory it will use to store this file with DOM parser in memory is also 1.5G or more? – michael Jul 17 '16 at 02:28
  • @michael: that depends on the structure of the XML document. I would expect a document with lots of nested elements to require more memory to parse than a single element with 1.5GB of text. For the record, I've seen a DOM parser use 400MB of memory for a 10MB XML file, so your XML file might require 60GB of memory. – Luke Woodward Jul 17 '16 at 10:42
  • @LukeWoodward Thanks for your answer. This make sense! – michael Jul 19 '16 at 05:11

1 Answers1

0

I wonder why there is clearly huge space not used, i.e. "8096m - 80m ", but still get to out of memory.

You are misinterpreting something1. And (IMO) the most likely think is that you are confusing the memory used by your IDE (Eclipse) with the memory used to run your application.

Unless you are doing something strange (and inadvisable) your application is run in a separate JVM whose heap parameters are independent of the IDE heap parameters. Unsurprisingly (to me) actual the memory usage of your Eclipse IDE process is small ... because that's not where your application was running.

Tweaking the heap parameters for Eclipse makes no difference to the application JVM heap parameters! What you actually need to do is to set the heap parameters in the Eclipse launcher config for your application. Within Eclipse. Alternatively, launch the application by hand from the command prompt.

But in this case, I doubt that it will work. I would be very surprised if you could represent a 1.5 GB XML file as a DOM in an 8GB Java heap. The overheads of that form of representation are too large.

1 - If the Eclipse IDE itself was running out of memory, it would get horribly slow, and then crash. Been there. Seen that. It typically happens if you are developing or examining a very large code-base.

Stephen C
  • 698,415
  • 94
  • 811
  • 1,216
  • Thanks for your solution that make sense! I still have two confused things: 1. how to set the jvm memory in eclipse, could you be more specific 2. what's the differencce for jvm and eclipse memory, the variables stored in jvm memory, then what eclipse memory used for? – michael Jul 17 '16 at 02:25
  • 1) http://stackoverflow.com/questions/15313393/how-to-increase-application-heap-size-in-eclipse. 2) It is like you have two cars. They have two petrol tanks. You put petrol in one car's petrol tank, and it is not in the other car's petrol tank. The JVM that runs Eclipse is different to the JVM that runs your application. – Stephen C Jul 17 '16 at 03:04
  • Thanks Steven. two quick questions 1. Since I could monitor the eclipse memory use in eclipse, is there a way to monitor the jvm memory use. After I change the jvm memory to 4G, it also works out of memory, as you say, dom may not deal with big data. 2. I also not get a clear idea for the differencce for jvm and eclipse memory, could you give me a link or explain more, your analog is kind of confused. – michael Jul 17 '16 at 03:54