How to compress a file using java on hdfs

Question

I'm new to hdfs/hadoop and need to know how to compress a file that is in a hdfs dir like hdfs://sandbox:8020/some/path.

I have tried

      Path p = new Path("/my/path/test1.gz");
      FSDataOutputStream os = fs.create(p);

      GZIPOutputStream gzipOs = new GZIPOutputStream(new BufferedOutputStream(os));

      Path filePath = file.getPath();
      FSDataInputStream is = fs.open(filePath);

      System.out.println("Writing gzip");

      byte[] buffer = new byte[1024];
      int len;
      while((len= is.read(buffer)) != -1){
        gzipOs.write(buffer, 0, len);
      }
      //close resources
      is.close();
      gzipOs.close();

But it doesn't work.

Any suggestions? Thanks in advance.

Answer 1

Below code is from Tom White's Definitive guide.

public class StreamCompressor {
  public static void main(String[] args) throws Exception {

  String codecClassname = args[0];
  Class<?> codecClass = Class.forName(codecClassname);
  Configuration conf = new Configuration();
  CompressionCodec codec = (CompressionCodec)
  ReflectionUtils.newInstance(codecClass, conf);
  CompressionOutputStream out = codec.createOutputStream(System.out);
  IOUtils.copyBytes(System.in, out, 4096, false);
  out.finish();
 }
}