How to write more than 30 MB of data in xml?

Question

First of all sorry if I'm repeating this question but I don't find any relevant solutions for my problem.

I'm facing difficulty in finding the way to solve the below issues.

1) I'm facing a scenario where I have to write more than 30 MB - 400 MB of data in a xml. When I'm using 'String' object to append the data to xml I'm getting 'OutOfMemory' exception.

After spending more time in doing R&D, I came to know that using 'Stream' will resolve this issue. But I'm not sure about this.

2) Once I constructed the xml, I have to send this data to the DMZ server using Android devices. As I know sending large amount of data using Http is difficult in this situation. In this case,

     a) Using FTP will be helpful in this scenario?
     b) Splitting the data into chunks of data and sending will be helpful?

Kindly let me know your suggestions. Thanks in advance.

Splitting the file and/or compressing it (Huffman coding would be pretty good for an XML IMHO) will help a lot on the upload times — Machinarius, Mar 26 '13 at 04:11

score 2 · Answer 1 · edited May 23 '17 at 11:57

2

i would consider zipping up the data before ftping it across.You could use a ZipOutputStream .

For the Out of Memory Exception, you could consider increasing the Heap Size. Check this : Increase heap size in Java

Can you post some values of heap size you tried, your code and some exception traces?

edited May 23 '17 at 11:57

Community

1
1

answered Mar 26 '13 at 04:08

DntFrgtDSemiCln

1,259
2
16
35

yes i'll be doing that, thanks. Kindly let me know if you have any thoughts over using 'Stream' for 'OutofMemory' exception. – macOsX Mar 26 '13 at 04:11
Yeah, XML should compress down about a 10-to-1 ratio. – djangofan Mar 26 '13 at 04:21
I have increased the heap size of dalvik VM and tried. But still I'm getting the same error. – macOsX Mar 26 '13 at 04:30

score 1 · Answer 2 · answered Mar 26 '13 at 04:11

1

Use StAX or SAX. These can create XML of any size because they write XML parts they generate to OutputStream on the fly.

answered Mar 26 '13 at 04:11

Evgeniy Dorofeev

133,369
30
199
275

score 0 · Answer 3 · answered Mar 26 '13 at 04:53

What you should do is

First, use a XML parser to read and write data in XML format. it could be SAX or DOM. If data size is huge try CSV format it will take less space as you do not have to store XML tag.

Second, When creating output file make sure those are small small files.

third when sending over network, make sure you zipped everything.

And for god sake, don't eat up user mobile data cap for this design. Warn user about this file size and suggest him to use WiFi network.

How to write more than 30 MB of data in xml?

3 Answers3