2

I have been using Cassandra and its C++ driver to write APIs to insert and fetch data for some time now. However now that I've created my cluster, I want to develop monitoring tools for my cluster.

I want to build an application(preferably in C++ and I don't want to use a 3rd party application), which will store Cluster management specific attributes like memory utilization of each node in the cluster, latency of each operation, space occupied by each table on each node etc. I read about 'Metrics in Cassandra(https://cassandra.apache.org/doc/latest/operating/metrics.html) but I don't know how exactly to use them in building my application as I've not worked on Java before(excuse me for that!). Can such application be built using C++? If it's a lot of work in C++, then it will be highly beneficial if you can share some Java code where these Cassandra Metrics have been used to monitor a Cassandra Cluster.

OS: RHEL Cassandra version: 3.11.2

Vishal Sharma
  • 1,670
  • 20
  • 55
  • Why do you want to build this one in C++. That actually adds some complexity to this task. If you have your cluster under control you could use somethink like jolokia which offers a HTTP interface for the MBeans cassandra uses to expose it's metrics. – TobiSH Mar 23 '18 at 09:12
  • Actually C++ is the only programming language I've worked on till now. So that's why it would be a bit easier for me I think. – Vishal Sharma Mar 23 '18 at 09:14
  • 1
    Metrics seem to be designed for Java in first place. For C++, you might dig a little deeper into the [http](http://metrics.dropwizard.io/3.1.0/getting-started/#reporting-via-http) interface... – Aconcagua Mar 23 '18 at 09:40
  • In the lib folder inside Cassandra's directory, there are 3 jar files(which seem to be related to Cassandra Metrics).. metrics-core-3.1.0.jar, metrics-jvm-3.1.0.jar, metrics-logback-3.1.0.jar...I've also found out about a yaml file in the conf directory "metrics-reporter-config-sample.yaml". But I still don't know how all of these work together – Vishal Sharma Mar 23 '18 at 11:18
  • Before you get too far down the rabbit hole. Fire up jconsole and point it to your server. You may need a username and password but this will show you what metrics are available. – Chris Hinshaw Mar 27 '18 at 05:59

2 Answers2

2

Cassandra 3.x uses the drop wizard api as you alluded to. If you can add the Jolokia jars to your deployment server(s) this will allow you to access the java jmx data using a simple http request. Jolokia exposes all the mbeans from java over a rest api.

Chris Hinshaw
  • 6,967
  • 2
  • 39
  • 65
1

It seems, there are no any c++ libs for JMX, but in Java it is pretty easy to get JMX metrics, all of you need is standard jdk. The following code demonstrates how to connect to cassandra node and get 'down' node count.

import java.util.HashMap;
import java.util.Map;

import javax.management.MBeanServerConnection;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;

public class Main {
    public static void main( String[] args ) throws Exception {
        String node = args[0];
        String port = args[1];
        String username = args[2];
        String password = args[3];

        JMXServiceURL url = new JMXServiceURL( "service:jmx:rmi:///jndi/rmi://" + node + ":" + port + "/jmxrmi" );
        String[] credentials = {username, password};

        Map<String, String[]> environment = new HashMap<>();
        environment.put( JMXConnector.CREDENTIALS, credentials );
        JMXConnector jmxConnector = JMXConnectorFactory.connect( url, environment );
        MBeanServerConnection mbsc = jmxConnector.getMBeanServerConnection();//Get metrics bean
        ObjectName oName = new ObjectName( "org.apache.cassandra.net:type=FailureDetector" );//create JMX object name
        int downNodes = (int) mbsc.getAttribute( oName, "DownEndpointCount" ); //get number of unavailable nodes

        System.out.println("Down node count: " + downNodes);
    }
}

More details about jmx you can find in Oracle documentation

To obtain JMX object names and attribute names you can use jconsole tool, which is shipped together with jdk:

enter image description here

Mikhail Baksheev
  • 1,394
  • 11
  • 13
  • As I had mentioned, Java is new to me. I had heard that using JMX is slow. I don't know how true that statement is but is there any other method(a faster method) where we don't use JMX and can still monitor Cassandra? – Vishal Sharma Mar 29 '18 at 05:34
  • https://cassandra.apache.org/doc/latest/operating/metrics.html says that "metrics can be queried via JMX or pushed to external monitoring systems using a number of built in and third party reporter plugins." I wanted to know whether these 'built in and third party reporter plugins' are also using the JMX Mbeans or they are collecting the metric values via some other method(which might be faster)? – Vishal Sharma Mar 29 '18 at 05:36
  • As @Chris Hinshaw mentioned, Cassandra uses dropwizard to collect metrics and as I know it does't use JMX to collect metrics, but I can be wrong. Dropwizard has two interfaces for reporting metrics: jmx and http, but which one is faster I can't say. I have an experience with jmx on production environment. We collect metrics from cassandra nodes via jmx by using zabbix agents, and we haven't faced any performance issues with this. – Mikhail Baksheev Mar 29 '18 at 07:06
  • In this link, http://metrics.dropwizard.io/3.1.0/manual/core/ , there's a warning that recommends not trying to gather metrics from production environment as JMX’s RPC API is fragile! – Vishal Sharma Mar 29 '18 at 09:27
  • As I understand, JMX can be a problem if you have a lot of jmx connections from many clients or high frequency of jmx calls. In our production we use only one client (zabbix agent) for each cassandra node, which request metrics once per minute. Users monitor metrics via zabbix ui, not directly via jmx. It works good for us. But if you want to built some tool using c++ , use a http reporter: https://www.datastax.com/dev/blog/pluggable-metrics-reporting-in-cassandra-2-0-2 – Mikhail Baksheev Mar 29 '18 at 10:24
  • Is it also possible to directly use the dropwizard metrics without using JMX, in Java? Are there APIs which I can use in my Java program(not the JMX method), which I'll call to fetch the metrics? Basically I was exploring ways of fetching the metrics and pushing them to some other server in JSON(I don't want to use JMX). – Vishal Sharma Mar 30 '18 at 05:57
  • There are two ways: 1. Use http dropwizard reporter (via metrics-servlets). I have no experience with this, but I guess you should place metrics-servlet jar into lib cassandra directory and configure it. Implement http client which will get metrics from the servlet. 2. Implement your own custom reporter for dropwizard and put it into lib cassandra directory. – Mikhail Baksheev Mar 30 '18 at 08:55
  • http://metrics.dropwizard.io/4.0.0/apidocs/com/codahale/metrics/servlets/MetricsServlet.html seems to be what I need but I don't know how exactly to build a code using this! – Vishal Sharma Mar 30 '18 at 09:01
  • you can download the jar from maven repo https://mvnrepository.com/artifact/io.dropwizard.metrics/metrics-servlets Choose the same version as another metrics jars in your cassandra lib directory – Mikhail Baksheev Mar 30 '18 at 10:28
  • Sorry, It was incorrect point, to work with metrics-servlet you need a servlet container. Cassandra does not provide it, so metrics-servlet wil not work – Mikhail Baksheev Mar 30 '18 at 10:42
  • Is there no way metrics-servlet can be used in cassandra? Maybe we need to manually add a jar file? – Vishal Sharma Mar 30 '18 at 11:58
  • metrics-servlet require a servlet container (http server). I think It will be easier to implement your own dropwizard metrics reporter, than make to work together all of this components: cassandra, metrics-servlet and some servlet container. Or just use Jolokia, but it will work via jmx – Mikhail Baksheev Mar 30 '18 at 16:54