5

reading about NoSQL (http://nosql.eventbrite.com/), a movement aimed at encouraging the dropping of traditional relational databases in favor of custom, application-suited storage systems.

Intrigued by the idea of trying to write a small personal storage system (for the .net framework) as a learning pet project, what are you suggestions or useful links? Where to start? How to balance what's on the hard drive and what's in memory?

I think this could be an interesting opportunity to learn the insides of database inner work, but I really lack the most basic theory of it. Thanks.

pistacchio
  • 56,889
  • 107
  • 278
  • 420

6 Answers6

3

The NoSQL movement is aimed at huge scale systems, at sizes where the relational model truly breaks. Before you start writing your own storage I highly recommend understanding the relational model, as is one of the best documented and well understood domains in CS. Start with the Gray's and Reuter's Transaction Processing, this book explains everything there is to know about implementing a classic RDBMS. Next on your list should be the Readings in Database Systems, this is a collection of the most relevant scientific papers and articles.

Remus Rusanu
  • 288,378
  • 40
  • 442
  • 569
2

The SO question "Implementing a database — How to get started" has some usefull answers to your question!

Community
  • 1
  • 1
jdecuyper
  • 3,934
  • 9
  • 39
  • 51
2

Before you get going I would recommend looking into SQL Servers ability to store XML files as BLOB objects inside the relational database. Perhaps your storage system doesn't need to be "from scratch". It could be a hybrid on top of SQLs XML storage capability.

djangofan
  • 28,471
  • 61
  • 196
  • 289
2

Well it all depends on the app you are building.

For example, if your app just needs to persist a few hundred objects and cut through them in a few ways and doesn't care if stuff gets corrupt once in a while. You could potentially just use LINQ to query a List and persist the List to disk once in a while.

If you need anything that has the magic ACID properties, well its going to take tons of work.

If you need something that supports Transactions, its going to take tons of work.

If you need something that understands the ANSI-SQL, you are going to have to write a parser, which is lots of work.

Before embarking on writing any kind of database I think you should understand a lot of database theory, get a book, read it.

Sam Saffron
  • 128,308
  • 78
  • 326
  • 506
2

Take a look at the work done by the Prevayler guys. They make the observation that if you can fit the data in ram, most usage scenario's show much better performance, and a lot less writing of code, without a RDBMS. On the other hand the Google, Amazon guys show that for large amounts of data you do not want to use a RDBMS. As we're moving to 64-bit OS-es and pcs with lots of ram, RDBMS's are between a rock and a hard place.

Stephan Eggermont
  • 15,847
  • 1
  • 38
  • 65
1

Although this is a late response. There are a few basic scenarios you need to take into account before you do this even if you have prior knowledge in how the dbo and its engine works. 1. Is it for heavy storage? If so, then you need to fine tune the pages and work on a file format that does take so much tick time to load and retrieve.

  1. Does it need to handle many connection? Again the pages are important but also you may need to create an engine for a Service or an app based instanced working behind the scenes.

  2. Is it for application usage or web usage? If it is for the web, then really use MySql or MSSQL.

Do not opt for inline memory to as your db storage because that nullifies the purpose of a database. The database was create so that you can free up the memory and release the table object(s) after an amount of time giving that memory back to the system. If it is for light use, create a simple XML/custom file database system because you are not saving or altering mass amounts of data at a time. Better than that, use SQLite which is very well suited for that purpose. If it is for opensource or commercial use do not go with inline memory because you don't to force someone to meet a high memory requirement, memory cost money and some folks are still running 32-bit OS.

Rob
  • 11
  • 1