2

Can an XML file be used efficiently as a database to a VB.NET application?

I already coded it this way and the application is running successfully.

Question is: when the storage/data size increases, will the program run in the same way as it is running now? (i.e. with the same response times, etc.)

Oleg Vaskevich
  • 12,444
  • 6
  • 63
  • 80
Pooh
  • 105
  • 1
  • 3
  • 12
  • 1
    No, that's a pretty bad choice except for read-only (relatively small) data. To know how your app will work with more data, there's only one thing you can do: test it with more data. We have no idea how your app works. – Mat Nov 24 '13 at 18:39
  • Not sure what format are you using (winforms/web) but files (including XML files) are either for output purposes (result for the user) or a temporary/not-ideal alternative. Usually, the ideal alternatives are memory and, when the sizes increases beyond what memory can deal with, a database. Everything is relative: it would depend upon the exact conditions, constraints, etc.; but, in principle, if you can choose, better a database. – varocarbas Nov 24 '13 at 18:39
  • @varocarbas: winforms is der any db which supports tree structure, which is supported by vb.net and if it is free that would be great(Because using a paid version of DB will make the user pay more). – Pooh Nov 25 '13 at 07:58
  • There are lots of free DBs and as far as .NET is one of the most popular programming environments, it would be pretty unlikely finding one not working with. NET. Do some research. – varocarbas Nov 25 '13 at 08:27

3 Answers3

2

You will encounter many scalability issues by storing your data in an Xml file. I would suggest that you look at breaking your Xml into meaningful documents and storing these as in a database, either a traditional RDBMS, or in an Xml NoSql database.

  1. Multi User contention. Since you've tagged vb.net, your first problem may be concurrency and contention to the Xml file on the file system, before you encounter any other performance issues relating to the size of the database. Although multiple users may simultaneously read the file, you will have contention as soon as you have one or more concurrent writers.

  2. Lack of Random access / Index seeks - if you need to search within your data, using Xml will typically require continual scanning of the entire Xml file. This will be IO intensive. A dedicated database would be able to use data indexes to reduce IO.

  3. Memory - if the primary nature of access to the Xml data is read-only (e.g. if it contains predominantly static data used to populate screens etc, but not used for read/write data storage), you may be inclined to cache it in memory, to compensate for the lack of quick random access to the data caused by #2. This approach obviously has limitations.

Converting the file to a Database

You do not give specifics on what you are storing in the Xml file, however, if Xml is the most appropriate storage format, I would suggest you break the file into logical storage units (aggregate roots / documents) appropriate for your domain.

Storage of these documents can be done with either a dedicated Xml Document / NoSql database, or many of the traditional Sql databases also have good support for storage Xml documents (e.g. MS Sql Server).

StuartLC
  • 104,537
  • 17
  • 209
  • 285
  • The application is a hospital management sw. As xml is free I thought the cost can be reduced. Now if I change the backend- I have to change some modules(Thats fine.. not a problem). The problems are : 1) The size of the db i install to the users pc will be more than xml. the size of xml is in KBs. 2) Cost. 3) The coding is more suited to the tree structure of dataset now. Is there any DB which supports tree structure(Nodes)? – Pooh Nov 25 '13 at 07:53
  • And bro.. Considering the issue of Indexes- I used the parent node name as index :D – Pooh Nov 25 '13 at 08:32
  • There is a similar question [here](http://stackoverflow.com/questions/77726/xml-or-sqlite-when-to-drop-xml-for-a-database#77750). In process databases like SqlLite or SqlCE could be an option, although you will need to promote key to indexed row columns for seeks with SqlCE as it loses the Xml support of its bigger brothers. Sql Express has Xml Support, although would increase the installation footprint of your app. – StuartLC Nov 25 '13 at 12:46
  • Concurency is the only problem im facing if xml is shared- in networking of computers Bro. :/ :( rest all the problems r sorted out. :) vb.net supports handling of xml by importing system.xml – Pooh Nov 25 '13 at 16:08
1

For a Database XML is AWESOME, for DATA ONLY....

It's lightning fast too!

I created a module that allows me to store data super fast... On a 7200 RPM Drive with a 2.67 GHz 4 Core Intel i5 CPU

It post the following performance...

50,000 Entries Written in 0.18 Seconds...

20,000 Entries Pulled From Database in 0.17 Seconds...

The speed was NOT affected by the size of the database, & you can store unlimited amounts of data (e.g. terabytes), however, what you do with the data can slow down how fast you can actually post it, so if you are doing something like building a string, then that function will slow down the posting of the data, if you are just posting though, that's a different story...

Is that fast enough for you?

The problem is, it's rather difficult to set up such an efficient database, and sorry I cannot give my secrets away, as it took me a while to conceive how to store / pull / delete / update the data efficiently enough to blow SQL out of the freaking water...

Now, with that being said, you should know that it can be done, but the hard part is, are you willing to work that hard to create it yourself?

Gale_i
  • 11
  • 3
  • Now the size of the xml file is very small about 50KBs. The application is fast in adding the data, retrieving and deleting. The only problem I have found till now is concurency. – Pooh Nov 25 '13 at 08:05
  • loading into memory may be an option yes? – Gale_i Nov 25 '13 at 15:56
  • Concurency is the problem if xml is shared- in networking of computers. – Pooh Nov 25 '13 at 16:01
0

Show us the numbers...

We've got an in-house application that works perfectly well using a single 10Mb XML file as its database. But it's only doing about 10 transactions a day, so writing the whole 10Mb back to disk each time is perfectly viable. If it was 10 transactions a second, it would be a rather different story.

For higher data sizes and transaction rates, consider an XML database, which combines the scaleability of a database with the convenience and flexibility of XML.

Michael Kay
  • 156,231
  • 11
  • 92
  • 164
  • My application doesnt write the whole file back to the hard drive each time. It considers just the node each time. For adding a record- Adds some nodes into the file. For deleting Deletes just the selected node. For modification- Deletes the node and adds new node. – Pooh Nov 25 '13 at 08:08
  • What technology do you use for in-situ updates to an XML file? – Michael Kay Nov 25 '13 at 17:49
  • Sir, in case of modification I remove the node. and add the modified record to that place. Or we can assign values. node("NODE_NAME").innertext=value. this is what i do-> http://stackoverflow.com/questions/1295603/whats-a-good-example-to-write-xml-using-vb-net-2008 – Pooh Nov 26 '13 at 07:17
  • I don't know Linq but I strongly suspect that after you have made your update to the in-memory version of the document, it will rewrite the whole file back to disk. – Michael Kay Nov 26 '13 at 15:24
  • ooops... oh ya. u r right.. – Pooh Nov 30 '13 at 05:42