1

I'm building Cassandra 3x m1.large cluster on Amazon EC2. I've used DataStax Auto-Clustering AMI 2.5.1-pv, with Cassandra DataStax Community version 2.2.0-1.

When doing write benchmarks, on 'production' data, it seems that cluster can handle around 3k to 5k write requests per second, without read load. Nearly all the time nodes do:

  • Compaction of system.hints
  • Compaction of mykeyspace.mybigtable
  • Compaction of mybigtable index

However, what worries me is the low CPU usage. All of the 3 nodes have CPU usage ranging from 17% to 24% percents. Isn't the CPU usage too low? Isn't that limiting my write speed? It could be 100% for me.

BTW. How can I check what's limiting (CPU, memory, network, discs) my write performance?

Here are some stats:

netstats

tpstats

top

iostat

Edit:

  • I'm inserting data nicely distributed around cluster
  • I'm using Consistency level of One
piotrwest
  • 2,098
  • 23
  • 35

3 Answers3

4

First of all, CPU isn't 20%. While the CPU system is at 20%, the user CPU is at around 70%. Here is the explanation between user CPU and system CPU: User CPU time vs System CPU time?

Secondly, iostat invoked without arguments isn't the best way of looking at disc usage. From: Basic I/O Monitoring on Linux

Without a specified interval, iostat displays statistics since the system was up then exits, which is not useful in our case.

For more comprehensive look at the system, use

  dstat -rcdgilmnps 60

dstat

Now we see clearly the averages from the last minute. CPU idle is at 1-4%, we have ~340 ios with 15M write speed.

Next usefull tool is nodetool cfstats: cfstats

Where we can see some stats for particular table. Write latency statistics are particularly interesting and equals to 1.5ms.

Finally, executing trace for write:

id: 12345 -> host NodeAsked:9042, achieved consistency: LocalOne
Sending MUTATION message to /NodeA on NodeAsked[MessagingService-Outgoing-/NodeA] at 0
Sending MUTATION message to /NodeB on NodeAsked[MessagingService-Outgoing-/NodeB] at 0
REQUEST_RESPONSE message received from /NodeA on NodeAsked[MessagingService-Incoming-/NodeA] at 0
Processing response from /NodeA on NodeAsked[SharedPool-Worker-32] at 0
MUTATION message received from /NodeAsked on NodeA[MessagingService-Incoming-/NodeAsked] at 12
Determining replicas for mutation on NodeAsked[SharedPool-Worker-45] at 114
Appending to commitlog on NodeAsked[SharedPool-Worker-45] at 183
Adding to mytable memtable on NodeAsked[SharedPool-Worker-45] at 241
Appending to commitlog on NodeA[SharedPool-Worker-5] at 5360
Adding to mytable memtable on NodeA[SharedPool-Worker-5] at 5437
Enqueuing response to /NodeAsked on NodeA[SharedPool-Worker-5] at 5527
Sending REQUEST_RESPONSE message to /NodeAsked on NodeA[MessagingService-Outgoing-/NodeAsked] at 5739

Shows that what's limiting us is the storage speed. It's the best to execute several spontaneous writes with enabled tracing on normal write load to see some patterns.

Vote up if you agree.

Community
  • 1
  • 1
piotrwest
  • 2,098
  • 23
  • 35
1

Is the application your using to benchmark available (open source) anywhere? Theres a chance if your application is doing something like serially sending requests that your throughput could be bottlenecked on latency (littles law) over the actual limits of your cluster. CPU should be the limiting factor in write performance so 20% does have the single threaded application look to it.

Theres a tool cassandra-stress that can mimick most kinds of loads which will fully utilize your client.

Chris Lohfink
  • 16,150
  • 1
  • 29
  • 38
  • It's my application which loads data from SQL DB, transforms it and puts it in SQL. It's very coupled to the business, so most probably I won't be open sourcing it. However, I've tested multiple load types (aka stress testing and load testing). Good point about cassandra-stress - I will try to run it at Monday. – piotrwest Aug 07 '15 at 14:31
  • worth trying, a lot of times the client end peaks out before the cluster does – Chris Lohfink Aug 07 '15 at 16:27
  • To use cassandra-stress with your data model try this tool www.sestevez.com/sestevez/CASTableSizer/ www.datastax.com/dev/blog/improved-cassandra-2-1-stress-tool-benchmark-any-schema – phact Aug 07 '15 at 18:27
  • Cassandra-stress tool with my schema showed the same level of performance. CPU also stayed at the same level. However, I've discovered something else - see my answer in a moment. – piotrwest Aug 10 '15 at 13:36
0

It is a matter of consistency. When you insert data, and consistency level is Quorum in your case, the driver waits until all nodes respond that the data is available, while inserting, do a Consistency of One, which will give you better performance. As for the performance of compaction please see following article: http://www.datastax.com/dev/blog/ec2-series-doc

Another reason why your writes are performing badly might be table design. If you do not set proper partition keys (depending on your data) then you might get long rows, which in general take longer when being compacted. If you want you can provide your table model (schema) and a sample of your data so this question might be answered in more detail.

Also bear in mind that C* was designed to run on commodity hardware. It rarely uses system resources to the full extent of i.e. available processor power. Cassandra can however - on reads - leverage as much memory as you feed it! As for the write throughput, there is a tool called CCM (https://github.com/pcmanus/ccm) which can benchmark your installation...

pcdoc
  • 96
  • 1
  • 3
  • Thank you for answer! I've read those resources. However, I wasn't looking for explanation "why is it slow", I was looking for explanation why CPU is only 20%? Shouldn't it be higher? Wouldn't that yield better write speed? – piotrwest Aug 07 '15 at 13:38
  • The Processors of your machine are not storing your data, the Storage system is. So having high processor usage would be a bad thing, meaning something would be done on this level. When it comes to Cassandra, it's all about how your storage performs, and - in some cases tweaking your sets of nodes. A good start point is here: https://academy.datastax.com/ and in specific DS201 which will give you all the intel on what can be tuned and in which occasion to tune :) – pcdoc Aug 07 '15 at 13:45
  • ccm is designed for testing operations and features, it is not recommended for any kind of benchmarking – Chris Lohfink Aug 07 '15 at 14:07
  • @pcdoc probably meant cassandra-stress tool. Anyway, iostat shows usage around 1.4MB/s - this shouldn't be much for EC2 m1.large. – piotrwest Aug 07 '15 at 14:39