Java Split XML file

Question

How can i split an long XML-file into pieces with each a predefined different name?

Example this is my XML file pasted in one long XML, generated for testing. Now i have to split on envelope, each a new file.

<envelope>
 <tag1>1</tag1>
 <tag2>2</tag2>
 <tag3>3</tag3>
</envelope>
<envelope>
 <tag1>1</tag1>
 <tag2>2</tag2>
 <tag3>3</tag3>
</envelope>
<envelope>
 <tag1>1</tag1>
 <tag2>2</tag2>
 <tag3>3</tag3>
</envelope>

I have already work with splits before just not like this where there is no begin and end tag for the entire xml.

possible duplicate of [Split 1GB Xml file using Java](http://stackoverflow.com/questions/5169978/split-1gb-xml-file-using-java) — Joachim Sauer, Jun 10 '11 at 11:59
There are **tons** of duplicates of this question. Searching for 2 minutes would have found the answer. — Joachim Sauer, Jun 10 '11 at 12:00
Yeah but unlike them i don't have a begin and end tag for the XML. — Eve, Jun 10 '11 at 12:07
@Eve: in that case your input is not XML: An XML document **must** have a **single** root element. — Joachim Sauer, Jun 10 '11 at 12:09
If everything else is well-formed XML, then using a filtering `InputStream` that adds a start- and end-tag could be a good solution. — Joachim Sauer, Jun 10 '11 at 12:10
A sensible thing would be to verify if the generation process itself can be made to write the output into separate files. That way you don't have to process the output file. — Vineet Reynolds, Jun 10 '11 at 12:14
@Joachim its seems your comment beat me to the answer, if you like add an answer and I'll remove mine. — Simeon, Jun 10 '11 at 12:14

score 4 · Accepted Answer · answered Jun 10 '11 at 13:59

I suggest making it well formed and then using one of the SAX or StAX solutions as suggested. The only difference is that I would avoid loading the whole thing into memory and instead inject the start and end elements by way of a SequenceInputStream.

for example:

InputStream in = new SequenceInputStream(
                        // start doc
                        new ByteArrayInputStream("<root>".getBytes()),
                        new SequenceInputStream(
                           new FileInputStream("envelopes.txt"),
                           // end doc
                           new ByteArrayInputStream("</root>".getBytes())));

score 2 · Answer 2 · edited May 23 '17 at 12:12

2

As Joachim said this is not an XML.

You can try to add a root element programmaticly, save the file as a temp somewhere and then refer to the other similar question on how to split it.

Answering the comment:

This might help you load it. I doubt you should worry about the size, since to split it you'd have to load it in memory anyway and then write it again.

Then something like:

final String xmlWithRootElement = "<root>" + IOUtils.toString(yourFile) + "</root>";

should do it. (without so many hardcoded strings)

One last thing.

I would suggest finding a solution that works. Then if you're unhappy with the performance you can look for ways to optimize it or you can ask a performance related question.

edited May 23 '17 at 12:12

Community

1
1

answered Jun 10 '11 at 12:13

Simeon

7,582
15
64
101

I would like to do that but the xml file is alot bigger it has 1000 "envelope" with also contains 50 lines. so adding it would be a bit too much – Eve Jun 10 '11 at 12:21
1000 envelope elements is not a lot, is actually quite few IMO. If you had a 1000000 envelope elements than you might notice it. How big is the file ? – Simeon Jun 10 '11 at 12:45

score 0 · Answer 3 · answered Jun 10 '11 at 12:33

0

How about just read the file character by character and identify <envelope> and </envelope> sequences. Whenever you encounter <envelope> you start capturing to a buffer until you reach </envelope>. This way the file can be as big as the filesystem allows. XML manipulation on large files is a headache :-)

answered Jun 10 '11 at 12:33

Karl-Bjørnar Øie

5,554
1
24
30

What do you mean by 'is a headache' ? – Simeon Jun 10 '11 at 12:35
@Simeon: XML parsing has a huge overhead, and in this example the input is not wellformed XML, so you need to work around it. – Karl-Bjørnar Øie Jun 10 '11 at 12:36
This http://stackoverflow.com/questions/6204827/xml-parsing-too-slow/6205624#6205624. I have parsed a 150mb XML wih SAX (unfortunately...) and it took no more then 5 seconds. – Simeon Jun 10 '11 at 12:46
@ZeroPage indeed. Then what are we discussing ? :) OP said 'it has 1000 "envelope"' – Simeon Jun 10 '11 at 13:02
I have no idea what you are discussing, the word 'headache'? :) – Karl-Bjørnar Øie Jun 10 '11 at 13:03

Java Split XML file

3 Answers3