Convert Avro in to Parquet format

Question

I want to export data from database and convert in to Avro + Parquet format. Sqoop support Avro export but not Parquet. I try to convert the Avro object to Parquet using Apache Pig, Apache Crunch etc but nothing working out.

Apache pig gives me "Caused by: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist". But the input path exist on that location.

Apache Crunch always throw :java.lang.ClassNotFoundException: Class org.apache.crunch.impl.mr.run.CrunchMapper not found" despite I added it in to the Hadoop lib path.

What is the best and easy way to export data from DB in to Parquet format?

score 3 · Answer 1 · answered May 09 '14 at 20:39

3

I use Hive.

Create an external table on the Avro data. Create an empty Parquet table.

And then insert overwrite table PARQUET_TABLE select * from AVRO_TABLE.

Super easy :)

answered May 09 '14 at 20:39

Gwen Shapira

4,978
1
24
23

Does this let us specify any of the parquet options through Hive? – Alex McLintock Nov 05 '14 at 11:47
Actually, I'm not sure - which Parquet options you want to specify? – Gwen Shapira Nov 06 '14 at 00:59

score 2 · Answer 2 · answered Jan 04 '15 at 07:44

2

The most recent sqoop (1.4.6 I think) supports import to files containing data in Parquet format and also import to Parquet with associated Hive table creation.

answered Jan 04 '15 at 07:44

Ted Dunning

1,877
15
12

score 0 · Answer 3 · answered Aug 16 '14 at 21:10

I was able to dump a mysql table using sqoop1 into an avro file and then convert the avro file into a parquet file using avro2parquet https://github.com/tispratik/avro2parquet conversion tool. Once it was in parquet, i could upload it to hdfs and create a hive table on top of it. You need a parquet plugin in hive if running hive version prior to 0.13. Hive supports parquet natively in 0.13.

Convert Avro in to Parquet format

3 Answers3