Big Data, How to parse a huge xml file faster?

Question

I have a huge bible data that is in xml format. I am making an android Bible application. But I feel like my data is very huge.

In my research, I read that xml parser parses through the whole file till it gets the tag that it needs. Does anyone know an easier and faster way to parse all the data.

parse it in chunks? ie. if you have to book tag, skip the other tags, and parse from there? — SQLiteNoob, Jun 11 '16 at 18:14
@SQLiteNoob thanks for the quick reply. Can you please advice me on which method to use? — Beraki, Jun 11 '16 at 18:21
I think the key question is: what do you want to do with the data? Parsing is only a means to an end. — Michael Kay, Jun 12 '16 at 09:36

score 1 · Answer 1 · answered Jun 11 '16 at 21:29

You should use a SAX parser, it's the best way to parse large XML files. For instance you can do this:

         File inputFile = new File("input.txt");
         SAXParserFactory factory = SAXParserFactory.newInstance();
         SAXParser saxParser = factory.newSAXParser();
         UserHandler userhandler = new UserHandler();
         saxParser.parse(inputFile, userhandler);

score 1 · Accepted Answer · edited May 23 '17 at 12:16

SAX parsing may be appropriate when the data extraction logic is relatively simple and forward only... if you want to have the ease and comfort of traversing the hierarchical structure or XPath, then you are out of luck...

JDOM or DOM have serious memory usage issues...

VTD-XML is a library that spans the use cases too complicated for SAX StAX, and too memory intensive for DOM or JDOM.

While VTD-XML loads everything in memory, the memory footprint is a modest 1.3x~1.5x the size of the XML document, which is 3~5x more efficient than DOM..

It also exports a DOM like cursor API and supports XPath 1.0...

Can SAX Parsers use XPath in Java?

Big Data, How to parse a huge xml file faster?

2 Answers2