0

My application needs to use data in a XML file which is up to 5 GB in size. I Load data in Image Classed from the XML. The Image class has many attributes, Like Path, Name, MD5, Hash, and many other information like that.

The 5 GB file has around 50 Million of Image data in it, When i parse the xml the data is loaded inside the app and same amount of image classes is created inside the app, and i perform different operation and calculation on it.

My Problem is when i parse such a hugh file my memory get eat up. I guess all the data is loading inside the ram. Due to complexity of the code, I'm unable to provide the whole code. I there an efficient way to handle such a hugh number of classes. I have done research all night, but didn't get success, Can some one point me in right direction?

Thanks

Community
  • 1
  • 1
Aqeel Haider
  • 603
  • 7
  • 24
  • I assume that you know not to use a DOM parser but instead to use a streaming parser such as SAX or StAX, right? But you don't mention this -- why? How exactly **are** you parsing the data? – Hovercraft Full Of Eels Jun 12 '17 at 01:18
  • I using SAX parser, but because of such a hugh amount of data, 50 million image objects the app memory eats up, if i increase the memory the app works fine. The app data can also be load from sqlite, i get same problem from there also, the problem is such a hugh amount of data, and i guess such a hugh amount of objects all are inside the RAM. – Aqeel Haider Jun 12 '17 at 01:24
  • Yes, after parsing the xml the data is saving inside the sqlite if i want to use the data in future, i can laod from sqlite which is much faster then parsing the xml, but problem comes when the amount of data exceed, the image class has important parameters, like image path, name, so i have to load all the data back in the app, which create same number of objects again, – Aqeel Haider Jun 12 '17 at 01:29

2 Answers2

0

You need some sort of pipeline to pass the data on to its actual destination without ever store it all in memory at once

I don't know how your code doing the parsing but you you don't need to store all data in the memory.

Here is a very good answer for implementation for reading large XML files

Fady Saad
  • 1,169
  • 8
  • 13
0

If you're using SAX, but you are eating up memory, then you are doing something wrong, and there is no way we can tell you what you are doing wrong without seeing your code.

I suggest using JVisualVM to get a heap dump and see what objects are using up the memory, and then investigating the part of your application that creates those objects.

Michael Kay
  • 156,231
  • 11
  • 92
  • 164