How to decompress lzo_deflate file?

Question

I used LZO to compress reduce output. I tried this: Hadoop-LZO project of Kevin Weil and then used LzoCodec class with my job:

TextOutputFormat.setOutputCompressorClass(job, LzoCodec.class);

Now compression works just fine.

My problem is that compression result is a .lzo_deflate file which I just can't decompress.
Lzop utility doesn't seem to support that type of file.
LzopCodec is supposed to give a .lzo file, but it did not work, however it's in th same package as LzoCodec (org.apache.hadoop.io.compress) which may refer to a compatibility issue, since I used the old API (0.19) to make compression works.

Answers to this question suggest Python solutions, however I need it in Java.
I'm using Hadoop 1.1.2 and Java 6.

What do you mean by "it did not work" for LzopCodec? LzopCodec is recommended over LzoCodec, it should be working. Can you include the error you have using that? — Charles Menguy, May 21 '13 at 18:44
Yes.`Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/io/compress/LzopCodec`. I tried to fix the issue until I read somewhere that LzoCodec is more recommended. So that should be clear before. — Nadjib Mami, May 21 '13 at 18:52
The big difference is that Lzop adds headers while Lzo doesn't. Have you updated your hadoop-env.sh and set the HADOOP_CLASSPATH and JAVA_LIBRARY_PATH correctly? — Charles Menguy, May 21 '13 at 18:58
Yes I have. I also commented out the `JAVA_LIBRARY_PATH = ''` in the `/path/to/hadoop/bin/hadoop` file. I checked the lzop library using `/path/to/hadoop/bin/hadoop classpath` command, lzop lib is there the last one. It should work like LzoCodec worked. Have you any idea @CharlesMenguy ? — Nadjib Mami, May 21 '13 at 21:05
I also tried to execute the two exports (`export HADOOP_CLASSPATH=` and `export JAVA_LIBRARY_PATH=`) through the command line, but the same thing. — Nadjib Mami, May 21 '13 at 21:11

score 7 · Answer 1 · answered May 21 '13 at 20:48

7

.lzo_deflate means an LZO stream without the usual header and trailer. So you would need to wrap the raw .lzo_deflate stream with the header and trailer expected by lzop. Or at least the header, and then ignore errors from the missing trailer. You'll need to look at the header and trailer documentation.

The "deflate" in the name is an odd choice, but it refers to the gzip analogy, where the raw compressed data format without the gzip header and trailer is called deflate.

answered May 21 '13 at 20:48

Mark Adler

101,978
13
118
158

Thanks @Mark Adler for your answer. I understand more now. – Nadjib Mami May 21 '13 at 21:08
1

Can't we decompress it via command line using the "lzop" tool? I am getting an error like "not a lzop file". – Sujai Sivasamy Feb 06 '19 at 09:50

score 4 · Answer 2 · answered Oct 23 '14 at 21:18

4

I came across the same issue. The reason it happened because I was not using the right codec. Please check your codec in job configuration.

job.getConfiguration().set("mapred.output.compression.codec","com.hadoop.compression.lzo.LzopCodec");

answered Oct 23 '14 at 21:18

drftr08

59
5

Vishrant · Answer 3 · 2020-07-24T03:01:18.027

0

This answer helped me to convert from .lzo_deflate to required output format:

hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-0.20.2-cdh3u2.jar \
  -Dmapred.output.compress=true \
  -Dmapred.compress.map.output=true \
  -Dmapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec \
  -input <input-path> \
  -output $OUTPUT \
  -mapper "/bin/cat"

edited Jul 24 '20 at 03:01

answered May 21 '20 at 19:08

Vishrant

15,456
11
71
120

How to decompress lzo_deflate file?

3 Answers3