Hadoop HDFS Copy Files from multiple folders to one destination folder

Question

What is the effective way to copy smaller files from multiple HDFS folders to one destination folder? The smaller files also need to be merged for Map-Reduce to be effective.

Possible duplicate: http://stackoverflow.com/questions/3548259/merging-multiple-files-into-one-within-hadoop — ffriend, Aug 06 '13 at 13:05

score 0 · Answer 1 · answered Aug 06 '13 at 14:25

0

There is DistCp which is an map-reduce job which copies files from one or multiple source folders to one target folder in an parallel manner. However, its not merging files. But maybe you could use filecrush to do that! (let me know how this goes!)

answered Aug 06 '13 at 14:25

oae

1,513
1
17
23

score 0 · Answer 2 · answered Aug 07 '13 at 18:21

0

You can simply run default Map-Reduce job (with default Mapper and Reducer) with "multiple HDFS folders" as input and one destination folder as output.

answered Aug 07 '13 at 18:21

Obus

51
1
6

Hadoop HDFS Copy Files from multiple folders to one destination folder

2 Answers2