Low memory overhead XML Parsing

Question

Introduction:

I'm developing a web app (Grails) and have a XML operation (upload&import of >20MB file) that has a huge memory consumption causing OutOfMemoryExceptions in my productive environment.

So I want to compare XMLSlurper/XMLParser/XmlTwig in memory consumption (to be honest I'm too lazy to write a SAX parser for my DOM)

What I want to know:

I'm lacking a good idea how to start some batch executions in a non productive environment (plain groovy instead of grails environment). I want to test different implementations several times, measure memory consumption (and maybe cpu time) and displaying the average. It may also matter how the memory settings of my productive environment is set (-XX:MaxPermSize, ..., on Tomcat), it will be nice to change that also.

Has anyone an idea on how to do this?

Answer 1

This is a tough question to provide a direct answer to. But here I go...

Using a DOM parser or something like Groovy's XMLSlurper for a URL should do the trick (or upload the file to the server and then use XMLSlurper). Anytime you deal with large files you really want to avoid loading the entire thing up in memory and then doing work on it, and it looks like you understand why this is. Comparing each implementation seems like overkill so just pick one that you trust that doesn't load everything into memory up-front.

Additional, as far as jvm settings go for tomcat you should utilization the hardware that's available to you... within reason ;). This is especially true in a java environment where more memory is need to operate effectively so max out the settings (-Xmx2048m -Xms2048m -XX:MaxPermSize=256m).