Sending many get requests efficiently in hbase

Question

I'm trying to create a generic method in Java for querying hbase.

I currently have one written which takes in 3 arguments

• A Range (to scan the table)
• A Column (to be returned) ... and
• A Condition (i.e. browser==Chrome)

So a statement (if written in a SQLish language) may look like

SELECT OS FROM TABLE WHERE BROWSER==CHROME IN RANGE (5 WEEKS AGO -> 2 WEEKS AGO)

Now, I know I'm not using HBase properly (using common column queries for rowkey etc.) but for the sake of experimentation I'd like to try it, to help me learn.

So the first thing I do is set a Range on the Scan. (5 weeks to 2 weeks ago), since the rowkey is the timestamp, this is very efficient.

Then I set a SingleColumnValueFilter (browser = Chrome) (after the range filter, this is pretty fast)

Then I store all the rowkeys (from the scan) into an array.

For each rowkey (in the array) I perform a GET operation to get the corresponding OS.

I have tried using MultiGet, which sped up the process a lot.

I then tried using normal GET requests, each spawning a new thread, all running concurrently, which halved the query time! But still not fast enough.

I have considered limiting the number of threads using a single connection to the database. i.e - 100 threads per connection.

Given my circumstances, what is the most efficient way to perform these GETs, or am I totally approaching it incorrectly?

Any help is hugely appreciated.

EDIT (Here is my threaded GET attempt)

List<String> newresults = Collections.synchronizedList(new ArrayList<String>());

for (String rowkey : result) {
    spawnGetThread(rowkey, colname);
}

public void spawnGetThread(String rk, String cn) {
    new Thread(new Runnable() {
        public void run() {

            String rt = "";
            Get get = new Get(Bytes.toBytes(rk));
            get.addColumn(COL_FAM, cn);
            try {
                Result getResult = tb.get(get);
                rt = (Bytes.toString(getResult.value()));
            } catch (IOException e) {
            }
            newresults.add(rt);
        }
    }).start();
}

Answer 1

Given my circumstances, what is the most efficient way to perform these GETs, or am I totally approaching it incorrectly?

I would suggest the below way

Get is good if you know which rowkeys you can acccess upfront.

In that case you can use method like below , it will return array of Result.

/**
     * Method getDetailRecords.
     * 
     * @param listOfRowKeys List<String>
     * @return Result[]
     * @throws IOException
     */
    private Result[] getDetailRecords(final List<String> listOfRowKeys) throws IOException {
        final HTableInterface table = HBaseConnection.getHTable(TBL_DETAIL);
        final List<Get> listOFGets = new ArrayList<Get>();
        Result[] results = null;
        try {
            for (final String rowkey : listOfRowKeys) {// prepare batch of get with row keys
   // System.err.println("get 'yourtablename', '" + saltIndexPrefix + rowkey + "'");
                final Get get = new Get(Bytes.toBytes(saltedRowKey(rowkey)));
                get.addColumn(COLUMN_FAMILY, Bytes.toBytes(yourcolumnname));
                listOFGets.add(get);
            }
            results = table.get(listOFGets);

        } finally {
            table.close();
        }
        return results;
    }

Additional Note: Rowfilters are always faster than column value filters( Which does full table scan)..

Would suggest to go through hbase-the-definitive guide -->Client API: Advanced Features