19

I have started to look into Hadoop. If my understanding is right i could process a very big file and it would get split over different nodes, however if the file is compressed then the file could not be split and wold need to be processed by a single node (effectively destroying the advantage of running a mapreduce ver a cluster of parallel machines).

My question is, assuming the above is correct, is it possible to split a large file manually in fixed-size chunks, or daily chunks, compress them and then pass a list of compressed input files to perform a mapreduce?

Jørn Schou-Rode
  • 37,718
  • 15
  • 88
  • 122
Luis Sisamon
  • 191
  • 1
  • 1
  • 3

4 Answers4

6

BZIP2 is splittable in hadoop - it provides very good compression ratio but from CPU time and performances is not providing optimal results, as compression is very CPU consuming.

LZO is splittable in hadoop - leveraging hadoop-lzo you have splittable compressed LZO files. You need to have external .lzo.index files to be able to process in parallel. The library provides all means of generating these indexes in local or distributed manner.

LZ4 is splittable in hadoop - leveraging hadoop-4mc you have splittable compressed 4mc files. You don't need any external indexing, and you can generate archives with provided command line tool or by Java/C code, inside/outside hadoop. 4mc makes available on hadoop LZ4 at any level of speed/compression-ratio: from fast mode reaching 500 MB/s compression speed up to high/ultra modes providing increased compression ratio, almost comparable with GZIP one.

Carlo Medas
  • 803
  • 9
  • 15
  • 3
    LZ4 is NOT splittable in Hadoop. The 4mc is a file format that uses LZ4, much like LZ4 has its own Frame format, and the 4mc file format is splittable. Its important to make this distinction: an actual .lz4 file is not splittable in Hadoop: https://issues.apache.org/jira/browse/HADOOP-12990. – Harsh J Apr 08 '16 at 18:50
5

Consider using LZO compression. It's splittable. That means a big .lzo file can be processed by many mappers. Bzip2 can do that, but it's slow.

Cloudera had an introduction about it. For MapReduce, LZO sounds a good balance between compression ratio and compress/decompress speed.

Victor
  • 347
  • 1
  • 6
  • 3
    LZO is not splittable alone. You must run a separate process to index the LZO files so that compressed blocks line correctly with input splits. See the tiny baby of a line at the very end of the page: https://github.com/kevinweil/hadoop-lzo – jerluc Aug 10 '12 at 00:20
  • 3
    @Luis But do keep in mind that LZO is GPL licensed so the regular terms and conditions apply. Another alternative will be to use Google's Snappy compression. [Google Snappy](http://code.google.com/p/snappy/) It's packaged by default with Hadoop (I use 0.20.x) and other ecosystem frameworks like Apache Flume, etc also understand it well by default. – arcamax Feb 28 '13 at 11:15
3

yes, you could have one large compressed file, or multiple compressed files (multiple files specified with -files or the api).

TextInputFormat and descendants should automatically handle .gz compressed files. you can also implement your own InputFormat (which will split the input file into chunks for processing) and RecordReader (which extract one record at a time from the chunk)

another alternative for generic copmression might be to use a compressed file system (such as ext3 with the compression patch, zfs, compFUSEd, or FuseCompress...)

jspcal
  • 50,847
  • 7
  • 72
  • 76
1

You can use bz2 as your compress codec, and this format also can been split.

zjffdu
  • 25,496
  • 45
  • 109
  • 159