Determine exactly files from HDFS to load into one Hbase table using java?

Question

I'm new at Big Data and Hadoop. I'm learning Hadoop and Hbase. I got a problem but still had no idea. Could you help me?

I've put 3 csv files to HDFS, include:
- File 1(Subscribe_info.txt): numID, active_date, status
- File 2(Recharge.txt): numID, recharge_history_date, amount, method
- File 3 (Charge.txt): numID, charge_date, amount_charge

All of them related with each other by: numID.

I need to load all data above into a Hbase table with row key is the numID and contain all columns related, like this:

Hbase table: rowkey= userID -- Column: active_date - status - recharge_history - amount_recharge - method - charge - history - amount

I use Java and I've created 3 classes to parse the data from 3 data files. But I don't know how I can read the file path from HDFS and parse to put it into the Hbase table.

Possible duplicate of [Loading csv data into Hbase](http://stackoverflow.com/questions/13906847/loading-csv-data-into-hbase) — Radim, Jul 14 '16 at 14:37
There are three different questions: how to read (and write) HDFS files? how to merge data from three files? how to import data into HBase? All three can be easily solved by following various tutorials. — Radim, Jul 14 '16 at 14:39

score 1 · Accepted Answer · edited May 23 '17 at 10:30

Your requirement has these steps

1) You have to read the HDFS file (in your case you have to read csv file you can use Open CSV just I have given an example below to read a normal file)

2) prepare a put and upload in to HBASE

I have already mentioned the code to do it here. pls have a look

import java.io.*;
import java.util.*;
import java.net.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;


public class Cat{
        public static void main (String [] args) throws Exception{
                try{
                        Path pt=new Path("hdfs://npvm11.np.wc1.yellowpages.com:9000/user/john/abc.txt");
                        FileSystem fs = FileSystem.get(new Configuration());
                        BufferedReader br=new BufferedReader(new InputStreamReader(fs.open(pt)));
                        String line;
                        line=br.readLine();
                        while (line != null){
                                System.out.println(line);
                                line=br.readLine();
                        }
                }catch(Exception e){
                }
        }
}

Hope this helps..

if you are okay with above answer, pls. flag-up as "accepted by owner" — Ram Ghadiyaram, Aug 04 '16 at 13:36

Determine exactly files from HDFS to load into one Hbase table using java?

1 Answers1