2

I am about to start developing a "time machine" for windows. I need to store a lot of records in a table. Every time I perform a backup I will write about 50,000 records because that's how many files I have in my computer (each records has nothing to do with storing a file. the record will contain the path where each file is located) Anyways I don't know if I could use XML to store my records since my queries are not going to be complicated plus it is easier to deploy versus creating a local database where I don't know if it is going to be more efficient because even though the queries are going to be simple I am going to be dealing with a lot of data. Moreover, I don't know if a local database can be bigger than 2GB and I am estimating that the database is going to be about 3 GB if regularly used for a year.

I prefer to use XML and use Xpath to do queries making it easier to deploy my application. But if creating a local database turns out to be more efficient and XML is going to slow my application because I have so many records then I will create a local database instead. Any other recommendations will be helpful maybe I need to do something else.

Alex Aza
  • 76,499
  • 26
  • 155
  • 134
Tono Nam
  • 34,064
  • 78
  • 298
  • 470

3 Answers3

6

Take a look at SQLite.

This is embedded yet powerful database. You don't need to install SQL server instance.
You can think of this database as a file with certain API. API allows you to run SQL queries against this file.

Alex Aza
  • 76,499
  • 26
  • 155
  • 134
  • +1 I've not used SQLite - but anything with a low footprint, decent API and good performance is a great idea. – Will A May 28 '11 at 04:58
  • I think a pointer to the C# binding would be helpful (and as I know nothing about C#, I can only guess this is correct: http://system.data.sqlite.org/index.html/doc/trunk/www/index.wiki – msw May 28 '11 at 05:02
2

SQL Server 2008 R2 Express allows databases of up to 10GB in size - so this may well be worth considering given your estimated space requirement.

It's much better to store your data in some form of database rather that in XML - as the latter will quickly grow to such a size that querying will become very slow and memory intensive.

Will A
  • 24,780
  • 5
  • 50
  • 61
  • 1
    To amplify: XML can be useful for data interchange or human readability. A blob of XML is not a database it is just a blob of xml, and is one of the least usable ways to store data for machine access and query. – msw May 28 '11 at 04:57
1

Using XML for storage of large amounts of data will require a lot of memory usage when you attempt to access the data. It is also not very efficient when it comes to space usage for storage. I would recommend going with a version of SQL. You could use one of the Express editions of Microsoft SQL or you could use an open source SQL flavor such as MySQL.

When you are looking at storing a lot of information, you need to look to a relational database. If you REALLY don't want to go this route, you could look at storing each backup in its own XML file. This would allow more portability and it would reduce your memory footprint when accessing the archived data.

IAmTimCorey
  • 16,412
  • 5
  • 39
  • 75