How to move HBase tables to HDFS in Parquet format?

Question

I have to build a tool which will process our data storage from HBase(HFiles) to HDFS in parquet format.

Please suggest one of the best way to move data from HBase tables to Parquet tables.

We have to move 400 million records from HBase to Parquet. How to achieve this and what is the fastest way to move data?

Thanks in advance.

Regards,

Pardeep Sharma.

"Parquet" you mean to say parquet avro ? Hbase is schema less where as parquet avro file has schema. what do you want to do with this data in parquet ? If you are using binary/protobuf these are the complex data types may create some issues while creating parquet. please see my answer. — Ram Ghadiyaram, May 04 '16 at 10:40
Yes it's Parquet avro. In our next step we'll use these parquet files for testing. Thanks for your immediate reply. — Pardeep Sharma, May 04 '16 at 11:50

score 1 · Answer 1 · answered Apr 06 '18 at 15:15

1

I recently opensourced a patch to HBase which tackles the problem you are describing. Have a look here: https://github.com/ibm-research-ireland/hbaquet

answered Apr 06 '18 at 15:15

Yiannis Gkoufas

Ram Ghadiyaram · Accepted Answer · 2017-10-18T08:47:42.747

0

Please have a look in to this project tmalaska/HBase-ToHDFS which reads a HBase table and writes the out as Text, Seq, Avro, or Parquet

hadoop jar HBaseToHDFS.jar ExportHBaseTableToParquet exportTest c export.parquet false avro.schema

edited Oct 18 '17 at 08:47

answered May 04 '16 at 10:42

Ram Ghadiyaram

Hi @RamPrasad G, Can we do the same in spark. – Pardeep Sharma May 05 '16 at 07:25
1

yes off course. There is a project called spark on Hbase(http://blog.cloudera.com/blog/2015/08/apache-spark-comes-to-apache-hbase-with-hbase-spark-module/). I think you use same input formats. Im not sure please check – Ram Ghadiyaram May 05 '16 at 07:43
I see that the project has not been updated for 3 years. Can anyone confirm if it still works? – Danny Feb 21 '17 at 20:47
Spark on Hbase is not supported for Spark 2.x versions. – rh979 Apr 15 '19 at 01:12

2 Answers2