0

How can we merge these n part files(output file from pig operation). I have to merge these files into one and then use that as an input in hive. I tried using -cat and -getmerge but no luck. Any suggestion will be helpful.

Thanks

Community
  • 1
  • 1
Purva Sharma
  • 33
  • 1
  • 3
  • Possible duplicate of [how to load multiple text files in a folder in pig using load command?](http://stackoverflow.com/questions/23622288/how-to-load-multiple-text-files-in-a-folder-in-pig-using-load-command) – Ani Menon Jan 25 '17 at 10:20

2 Answers2

0

You can specify the file in the below manner

A = LOAD '/hdfs/OUTPUT/ReliableProducts-r-*' using PigStorage('\t') as (Sid:int,Sname:chararray,Ttrading:chararray,Sloc:chararray,OBal:int,CBal:int,Frate:int);

Hope this Helps!!!

Bhavesh
  • 909
  • 2
  • 23
  • 38
0

The Hadoop FileUtil class has a copyMerge method that handles very well the multiple part files problem.

It will take all files in the folder you provide (although will not look recursively) and concatenate them all to produce a single file.

A.Perrot
  • 323
  • 1
  • 8