7

Note: (I have investigated CouchDB for sometime and need some actual experiences).

I have an Oracle database for a fleet tracking service and some status here are:

  1. 100 GB db
  2. Huge insertion/sec (our received messages)
  3. Reliable replication (via Oracle streams on 4 servers)
  4. Heavy complex queries.

Now the question: Can CouchDB be used in this case?

Note: Why I thought of CouchDB?

  1. I have read about it's ability to scale horizontally very well. That's very important in our case.
  2. Since it's schema free we can handle changes more properly since we have a lot of changes in different tables and stored procedures.

Thanks

Edit I: I need transactions too. But I can tolerate other solutions too. And If there is a little delay in replication, that would be no problem IF it is guaranteed.

Kaveh Shahbazian
  • 13,088
  • 13
  • 80
  • 139
  • Do you need transactions? Do you need guaranteed synchronous replication? – edze Apr 02 '12 at 08:38
  • Thanks for your clarification. Yes; I need transactions BUT I can tolerate some delay in replication. – Kaveh Shahbazian Apr 02 '12 at 08:50
  • 1
    http://stackoverflow.com/questions/299723/can-i-do-transactions-and-locks-in-couchdb – edze Apr 02 '12 at 09:30
  • Thanks edze! But this answered just one part! – Kaveh Shahbazian Apr 02 '12 at 14:39
  • Explain what you mean by "Heavy complex queries". Are they complex in a way that would fit the CouchDB model easier than a relational model? – Adam Hawkes Apr 02 '12 at 22:01
  • They fit the relational model perfectly. But - for some technical reasons - I am exporting/analyzing messages to xml files on a daily basis and use them for generating reports. I guess I can do the same with CouchDB. Also there are other real-time complex queries. But the point is since we are changing things too much, working with relational model is a pain. Besides Oracle streaming is buggy and stops for no obvious reason on it's own (So I had to write a service to run streaming again if it stops). – Kaveh Shahbazian Apr 03 '12 at 06:23
  • analyzing and aggregates with large data-sets. Did you test it with using materialized views? – edze Apr 03 '12 at 10:56
  • @edze I do not know materialized views; but thanks for the hint. As for now I am investigating PostgreSQL for it's replication abilities. I am getting this feeling that this kind of problem does not fit into CouchDB model. (For example - I have read recently about it - since CouchDB is an append only engine so I have to compact it periodically; that's a new problem!) – Kaveh Shahbazian Apr 03 '12 at 11:43
  • http://tech.jonathangardner.net/wiki/PostgreSQL/Materialized_Views – edze Apr 03 '12 at 12:49

2 Answers2

25

You are enjoying the following features with your database:

  1. Using it in production
  2. The data is naturally relational (related to itself)
  3. Huge insertion rate (no MVCC concerns)
  4. Complex queries
  5. Transactions

These are all reasons not to switch to CouchDB.

Of course, the story is not so simple. I think you have discovered what many people never learn: complex problems require complex solutions. We cannot simply replace our database and take the rest of the month off. Sure, CouchDB (and BigCouch) supports excellent horizontal scaling (and cross-datacenter replication too!) but the cost will be rewriting a production application. That is not right.

So, where can CouchDB benefit you?

I suggest that you begin augmenting your application with CouchDB applications. Deploy CouchDB, import your data into it, and build non mission-critical applications. See where it fits best.

For your project, these are the key CouchDB strengths:

  1. It is a small, simple tool—easy for you to set up on a workstation or server
  2. It is a web server. It integrates very well with your infrastructure and security policies.
    • For example, if you have a flexible policy, just set it up on your LAN
    • If you have a strict network and firewall policy, you can set it up behind a VPN, or with your SSL certificates
  3. With that step done, it is very easy to access now. Just make http or http requests. Whether you are importing data from Oracle with a custom tool, or using your web browser, it's all the same.
  4. Yes! CouchDB is an app server too! It has a built-in administrative app, to explore data, change the config, etc. (like a built-in phpmyadmin). But for you, the value will be building admin applications and reports as simple, traditional HTML/Javascript/CSS applications. You can get as fancy or as simple as you like.
  5. As your project grows and becomes valuable, you are in a great position to grow, using replication
    • Either expand the core with larger CouchDB clusters
    • Or, replicate your data and applications into different data centers, or onto individual workstations, or mobile phones, etc. (The strategy will be more obvious when the time comes.)

CouchDB gives you a simple web server and web site. It gives you a built-in web services API to your data. It makes it easy to build web apps. Therefore, CouchDB seems ideal for extending your core application, not replacing it.

JasonSmith
  • 72,674
  • 22
  • 123
  • 149
1

I don't agree with this answer..

I think CouchDB suits especially well fleet tracking use case, due to their distributed nature. Moreover, the unreliable nature of gprs connections used for transmitting position data, makes the offline-first paradygm of couchapps the perfect partner for your application.

For uploading data from truck, Insertion-rate can take a huge advantage from couchdb replication and bulk inserts, especially if performed on ssd-based couchdb hosting.

For downloading data to truck, couchdb provides filtered replication, allowing each truck to download only the data it really needs, instead of the whole database.

Regarding complex queries, NoSQL database are more flexible and can perform much faster than relation databases.. It's only a matter of structuring and querying your data reasonably.

giowild
  • 479
  • 5
  • 17