How to extract data from Hadoop sequence file?

Question

Hadoop sequence file is really strange. I pack the images into sequence file and can't recovery image. I do some simple test. And I found the size of bytes even not same before and after use sequence file.

Configuration confHadoop = new Configuration();
        FileSystem fs = FileSystem.get(confHadoop);

        String fileName = args[0];
        Path file = new Path(fs.getUri().toString() + "/" + fileName);
        Path seqFile = new Path("/temp.seq");
        SequenceFile.Writer writer = null;
        FSDataInputStream in = null;
        try{
            writer = SequenceFile.createWriter(confHadoop,Writer.file(seqFile), Writer.keyClass(Text.class),
                    Writer.valueClass(BytesWritable.class));

            in = fs.open(file);
            byte buffer[] = IOUtils.toByteArray(in);


            System.out.println("original size ---->  " + String.valueOf(buffer.length));
            writer.append(new Text(fileName), new BytesWritable(buffer));
            System.out.println(calculateMd5(buffer));
            writer.close();

        }finally{
            IOUtils.closeQuietly(in);
        }

        SequenceFile.Reader reader = new SequenceFile.Reader(confHadoop, Reader.file(seqFile));

        Text key = new Text();
        BytesWritable val = new BytesWritable();

        while (reader.next(key, val)) {
            System.out.println("size get from sequence file --->" + String.valueOf(val.getLength()));
            String md5 = calculateMd5(val.getBytes());
            Path readSeq=new Path("/write back.png");  
            FSDataOutputStream out = null;
            out = fs.create(readSeq);
            out.write(val.getBytes());  //YES! GOT THE ORIGIANL IAMGE
            out.close();
            System.out.println(md5);
            .............
}

The output shows I got the same number of bytes, and after I write the image back to the local disk, I am sure I got the original image. But why the MD5 value is not same?

What's wrong I made here?

14/04/22 16:21:35 INFO compress.CodecPool: Got brand-new compressor [.deflate]
original size ---->  485709
c413e36fd864b27d4c8927956298edbb
14/04/22 16:21:35 INFO compress.CodecPool: Got brand-new decompressor [.deflate]
size get from sequence file --->485709
322cce20b732126bcb8876c4fcd925cb

`in.available()` is the issue, do you know how to read a file? — Thomas Jungblut, Apr 22 '14 at 06:59
@ThomasJungblut I just change the way to read using this [answer](http://stackoverflow.com/a/1264756/1285444), and I can get the original image. The problem is the MD5s are still different. Sorry to trouble you. — hakunami, Apr 22 '14 at 08:32
What MD5's are different? Of two sequence files? The image and sequence file's MD5 must be different, because seq files contain checksums and sync points. — Thomas Jungblut, Apr 22 '14 at 09:55
@ThomasJungblut I find the solution finally, and post it as answer. Thank you for the advice courages me to find the solution by myself. :) — hakunami, Apr 23 '14 at 14:32

score 9 · Accepted Answer · answered Apr 23 '14 at 14:31

I finally solve this strange problem, and I have to share it. First, I will show you the wrong way to get the bytes from sequence.

Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
Path input = new Path(inPath);
Reader reader = new SequenceFile.Reader(conf, Reader.file(input));
Text key = new Text();

BytesWritable val = new BytesWritable();
    while (reader.next(key, val)) {
    fileName = key.toString();
    byte[] data = val.getBytes(); //don't think you have got the data!
}

the reason is getBytes() does not return the exactly size of your original data. I put the data in using

FSDataInputStream in = null;
in = fs.open(input);
byte[] buffer = IOUtils.toByteArray(in);

Writer writer = SequenceFile.createWriter(conf,
Writer.file(output), Writer.keyClass(Text.class),
Writer.valueClass(BytesWritable.class));

writer.append(new Text(inPath), new BytesWritable(buffer));
writer.close();

I check the size of output sequence file, it is original size plus head, I am not sure the reason why getBytes() give me more bytes than original. But let's see how to get the data correctly.

Option #1, copy the size of data you need.

byte[] rawdata = val.getBytes();
length = val.getLength(); //exactly size of original data
byte[] data = Arrays.copyOfRange(rawdata,  0, length); this is corrent

Option #2

byte[] data = val.copyBytes();

this is more sweet. :) Finally got it right.

Yes, they must of been on some acid trip when they thought `getBytes` was a good name for a method which DOES NOT GET THE BLOODY BYTES!!! Drove me round the bend trying to figure this one out. — samthebest, Jun 06 '14 at 13:35

How to extract data from Hadoop sequence file?

1 Answers1