REST API for processing data stored in hbase

Question

I have a lot of records in hbase store (millions) like this

key = user_id:service_id:usage_timestamp value = some_int

That means an user used some service_id for some_int at usage_timestamp.

And now I wanted to provide some rest api for aggregating that data. For example "find sum of all values for requested user" or "find max of them" and so on. So I'm looking for the best practise. Simple java application doesn't met my performance expectations.

My current approach - aggregates data via apache spark application, looks good enough but there are some issues to use it with java rest api so far as spark doesn't support request-response model (also I have took a view into spark-job-server, seems raw and unstable)

Thanks,

Any ideas?

Thank you for your answer! I just want to wait few days more for possible answers. Don't worry, I will accept some. — Normal, Oct 11 '16 at 00:15

score 1 · Accepted Answer · edited May 23 '17 at 12:00

I would offer Hbase + Solr if you are using Cloudera (i.e Cloudera search)

Solrj api for aggregating data(instead of spark), to interact with rest services

Solr Solution (in cloudera its Cloudera search) :

Create a collection (similar to hbase table) in solr.
Indexing : Use NRT lily indexer or custom mapreduce solr document creator to load data as solr documents.

If you don't like NRT lily indexer you can use spark or mapreduce job with Solrj to do the indexing For ex: Spark Solr : Tools for reading data from Solr as a Spark RDD and indexing objects from Spark into Solr using SolrJ.

Data Retrieval : Use Solrj to get the solr docs from your web service call. In Solrj,

There is FieldStatInfo through which Sum,Max etc.... can be achieved
There are Facets and Facetpivots to group data
Pagination is supported for rest API calls

you can integrate solr results with Jersey or some other web service as we have already implemented this way.

/**This method returns the records for the specified rows from Solr Server which you can integrate with any rest api like jersey etc...
     */
    public SolrDocumentList getData(int start, int pageSize, SolrQuery query) throws SolrServerException {
        query.setStart(start); // start of your page
        query.setRows(pageSize);// number of rows per page
        LOG.info(ClientUtils.toQueryString(query, true));
        final QueryResponse queryResponse = solrCore.query(query, METHOD.POST); // post is important if you are querying huge result set Note : Get will fail for huge results
        final SolrDocumentList solrDocumentList = queryResponse.getResults();
        if (isResultEmpty(solrDocumentList)) { // check if list is empty
            LOG.info("hmm.. No records found for this query");
        }
        return solrDocumentList;

}

Also look at

Note : I think same can be achieved with elastic search as well. But out of my experience , Im confident with Solr + solrj

@JustinKestelyn: Since you belong to cloudera Im asking this question :) What about cloudera search ? Is it opt for the above question. What option cloudera prefers for hbase + Spark kind of processing? — Ram Ghadiyaram, Oct 06 '16 at 18:53

score 1 · Answer 2 · edited Sep 19 '17 at 13:39

1

I see two possibilities:

Livy REST Server - new REST Server, created by Cloudera. You can submit Spark jobs in REST way. It is new and developed by Cloudera, one of the biggest Big Data / Spark company, so it's very possible that it will be developed in future, not abandoned
You can run Spark Thrift Server and connect just like to normal database via JDBC. Here you've got documentation. Workflow: read data, preprocess and then share by Spark Thrift Server

If you want to isolate third-party apps from Spark you can create simple application that will have user-friendly endpoint and will translate query received by endpoint to Livy-Spark jobs or SQL that will be used with Spark Thrift Server

edited Sep 19 '17 at 13:39

OneCricketeer

179,855
19
132
245

answered Oct 05 '16 at 10:59

T. Gawęda

15,706
4
46
61

Just FYI: Livy is not in fact "supported" by Cloudera for production use. But we do encourage its use for test & dev. – Justin Kestelyn Oct 05 '16 at 15:50
@JustinKestelyn You are right, I mean that Livy is developed by Cloudera, which means it will be developed in future, not abandoned :) – T. Gawęda Oct 05 '16 at 16:22
1

Sure, I just wanted to make that clear (the word "support" has specific meaning to paying customers). – Justin Kestelyn Oct 05 '16 at 17:02
@JustinKestelyn Thanks :) I've updated answer to not confuse anyone – T. Gawęda Oct 05 '16 at 17:25

REST API for processing data stored in hbase

2 Answers2