How to merge hive partitoned data in one large file?

Question

I have hive table partitioned on date and hour column. when I load the data i will create 24 files. I want merge this 24files in one file. Can anyone suggest me the solution

Answer 1

Well if you want to have one single file while inserting the data to your partitioned file then you can have your hive as follows:

• partitioned on date
• bucketed on any one column and have only 1 bucket.

Since your bucket number is 1, all your data will be in the file after insert.

Another way is to merge it using the hdfs commands like below

hadoop fs -cat hive_table_data_folder/p* > new_file_name