How to write a xml database file efficiently?

Question

I want to build an XML file as a datastore. It should look something like this:

<datastore>
    <item>
        <subitem></subitem>
        ...
        <subitem></subitem>
    </item>
    ....
    <item>
        <subitem></subitem>
        ...
        <subitem></subitem>
    </item>
</datastore>

At runtime I may need to add items to the datastore. The number of items may be high, so that I don't want to hold the whole document in memory and can't use DOM. I just want to write the part where a change occures. Or does DOM supports this?

I had a first look at StAX, but I am not sure if it does what I want.

Wouldn't it be the best to remember a cursor position at the end of the file just right before the root element is beeing closed? That is always the position where new items will be added. So if I remember that position and keep it up to date during changes, I could add an new item at the end, without iterating through the whole file .

Maybe a second cursor, could be used independendly from the first one, to iterate over the document just for reading purposes.

I can't see that StAX supports any of this, does it?

Isn't there a block based API for files instead of a stream bases one? Aren't files and filesystems typical examples for block "devices"? And if there is such an API, does it help me with my problem?

Thanks in advance.

Have you heard of vtd-xml and extended vtd-xml? they come as a sinle package. — vtd-xml-author, Jan 27 '11 at 20:05

score 1 · Answer 1 · answered Jan 26 '11 at 19:22

Updating XML is basically impossible because there's no "cheap" way to insert data.

Appending XML is not so bad. All you need to do there is seek to the end of the file, then GO BACK over the "end tag" (</datastore> in this case), and then just start writing. This is a cheap operation all told, but none of the frameworks really support this as they're all mostly designed to work with well formed, full boat XML documents, as a whole, not in pieces.

You could use a StAX like thing, but in this case, StAX isn't aware of the <datastore> tag, rather it's just aware of the <item> tags and its elements. Then you create Items and start writing, over and over and over, to the same OutputStream that you have set up.

That's the best way to do this.

But if you need to delete or change data, then you get to rewrite stuff, or do hacks, such as marking elements as "inactive", hunting them down in the XML file, seeking to the 'active="Y"' attribute, and then inplace changing the Y to N. It can be done, it will be mostly efficient, but its far and away outside what the normal XML processing frameworks let you do. If I were to do that, I'd read the entire file and keep track of those entries and note their locations within it so later I could easily seek and change them efficiently.

Then when you update something, you "inactivate" the old one, and "append" the new one. Eventually get to GC the file by rewriting it all and throwing out the old, "inactive" entries.

There is a cheap way to insert data, super cheap. VTD-XML is the only one that I am aware of right now. — vtd-xml-author, Jan 27 '11 at 20:07
That may well be, but in the end if you make an arbitrary internal change to the XML file, you will end up rewriting the entire XML file vs my suggestions which allow you to append to an XML file in place without rewriting it. Without special handling, adding 1K of data a 200MB XML file ends up writing 200MB of data to the disk. Using my append technique, you write 1K of XML to the disk. — Will Hartung, Jan 27 '11 at 20:24

biziclop · Answer 2 · 2011-01-26T23:29:16.017

As a rule of thumb, XML files aren't very efficient as datastores, not for the record-based data you seem to want to use them for.

But if you've already got the file and absolutely can't do anything about it, you can use StAX XMLEventReaders and XMLEventWriters to read through a file quickly and insert/modify elements in it.

But when I say quickly, what I mean is more quickly than DOM would be, but nowhere near as effective as any relational DB.

Update: Another option you can consider is vtd-xml, although I haven't tried it in real projects, it actually looks pretty decent.

score 0 · Answer 3 · answered Jan 26 '11 at 23:24

If you always want to append items at the end, then the best way to handle this is to have two XML files. The outer one datstore.xml is simply a wrapper, and looks like this:

<!DOCTYPE datastore [
  <!ENTITY e SYSTEM "items.xml">
]>
<datastore>&e;</datastore>

The file items.xml looks like this:

<item>....</item>
<item>....</item>
<item>....</item>

with no wrapper element.

When you want to append data, you can open items.xml and write to the end of it. When you want to read data, open datastore.xml with an XML parser.

Of course, once your data grows beyond 20Mb or so, it may well be better to use an XML database. But I've been using this approach for years for records of Saxon orders, with files that are currently about 8Mb, and it works fine.

score 0 · Answer 4 · answered Jan 26 '11 at 23:41

It's not very easy or efficient to partially update an XML file so you won't find much support for it as a use case.

Really it sound like you need a proper database, perhaps with a tool to export the data as XML.

If you don't want to use a DB and insist on storing the data purely as XML you might consider keeping all your items in memory as objects. Whenever a new one is added you can write all of them out to XML. It might seem inefficient, but depending on your data size might still be good enough.

If you choose this path, you might want to check out the Xstream library to make this quite easy, see stream tutorial for a quick example.

How to write a xml database file efficiently?

4 Answers4