15

I would like to hear your thoughts and experiences on the usage of CQL and in-memory query engine Spark/Shark. From what I know, CQL processor is running inside Cassandra JVM on each node. Shark/Spark query processor attached with a Cassandra cluster is running outside in a separated cluster. Also, Datastax has DSE version of Cassandra which allows to deploy Hadoop/Hive. The question is in which use case we would pick a specific solution instead of the other.

ChrisGPT was on strike
  • 127,765
  • 105
  • 273
  • 257
Minh Do
  • 329
  • 3
  • 7

2 Answers2

33

I will share a few thoughts based on my experience. But, if possible for you, please let us know about your use-case. It'll help us in answering your queries in a better manner.

1- If you are going to have more writes than reads, Cassandra is obviously a good choice. Having said that, if you are coming from SQL background and planning to use Cassandra then you'll definitely find CQL very helpful. But if you need to perform operations like JOIN and GROUP BY, even though CQL solves primitive GROUP BY use cases through write time and compact time sorts and implements one-to-many relationships, CQL is not the answer.

2- Spark SQL (Formerly Shark) is very fast for the two reasons, in-memory processing and planning data pipelines. In-memory processing makes it ~100x faster than Hive. Like Hive, Spark SQL handles larger than memory data types very well and up to 10x faster thanks to planned pipelines. Situation shifts to Spark SQL benefit when multiple data pipelines like filter and groupBy are present. Go for it when you need ad-hoc real time querying. Not suitable when you need long running jobs over gigantic amounts of data.

3- Hive is basically a warehouse that runs on top of your existing Hadoop cluster and provides you SQL like interface to handle your data. But Hive is not suitable for real-time needs. It is best suited for offline batch processing. Doesn't need any additional infra as it uses underlying HDFS for data storage. Go for it when you have to perform operations like JOIN, GROUP BY etc on large dataset and for OLAP.

Note : Spark SQL emulates Apache Hive behavior on top of Spark, so it supports virtually all Hive features but potentially faster. It supports the existing Hive Query language, Hive data formats (SerDes), user-defined functions (UDFs), and queries that call external scripts.

But I think you will be able to evaluate the pros and cons of all these tools properly only after getting your hands dirty. I could just suggest based on your questions.

Hope this answers some of your queries.

P.S. : The above answer is based on solely my experience. Comments/corrections are welcome.

Pooyan Khosravi
  • 4,861
  • 1
  • 19
  • 20
Tariq
  • 34,076
  • 8
  • 57
  • 79
  • 4
    Give a reason for downvote, i'll accept it with spirit..Just don't do this like a 5 year old. – Tariq Jun 15 '13 at 10:38
  • 1
    Apologies, it was unintentional. I didn't even notice I had done it until I checked in again today. – Gavin Jun 15 '13 at 14:51
  • -1 Spark SQL (formerly Shark) compiles SQL to RDD operations on top of Spark and provides facilities for compiling HiveQL to Spark's RDD operations rather than map and reduce operations on top of Hadoop. Also Spark isn't just in memory, it still handles datasets paged to disk 10 times faster. – Pooyan Khosravi Feb 02 '15 at 11:46
  • @PooyanKhosravi : Could you please point out where exactly in my answer I have written about **map and reduce operations on top of Hadoop** and **just in memory** thing? In fact, I have clearly specified that spark **runs on-disk queries, and they are still faster than Hive(by ~5-10x)**. – Tariq Feb 02 '15 at 13:37
  • 1
    I'm trying to avoid a discussion via comments @Tariq. Would an edit be okay to increase accuracy of your answer? I can write a new answer too. – Pooyan Khosravi Feb 02 '15 at 14:43
  • 1
    @PooyanKhosravi +1 for avoiding a discussion via comments. You can do whatever you like to make SO better. I just wanted to know about my mistake so that I can update the answer. BTW, downvote is not the way to suggest an edit. – Tariq Feb 02 '15 at 15:09
6

There is a very good effort for benchmark documented here - https://amplab.cs.berkeley.edu/benchmark/

Ambarish Hazarnis
  • 186
  • 1
  • 5