4

Main.java

import java.io.IOException;

public class Main
{

    private final CompressedOutputStream    m_cos;

    public static void main(String[] args)
    {
        try
        {
            final Main m = new Main(new CompressedOutputStream());
            m.run();
        }
        catch (IOException e)
        {
            e.printStackTrace();
        }
    }

    public Main(final CompressedOutputStream cos)
    {
        this.m_cos = cos;
    }

    private void trace(final String format, Object... args)
    {
        final String output = String.format(format, args);
        System.out.println(output);
    }

    public void run()
    {
        this.first_write();
        this.second_write();
        this.trace("CRC32 of data is %x", this.m_cos.get_crc32());
    }

    private void add_data(final byte[] data)
    {
        try
        {
            this.m_cos.write(data); // STEPPING INTO THIS IN DEBUG DOESN'T LEAD ANYWHERE!
        }
        catch (IOException e)
        {
            e.printStackTrace();
        }
    }

    public void first_write()
    {
        final String[] text_lines = new String[] { "Hello world!",
                "My name is Joe.", "Nice to meet.", "Where are you from?",
                "I'm from Congo" };

        for (final String text_line : text_lines)
        {
            final byte[] data = text_line.getBytes();
            this.add_data(data);
        }

        final byte[] compressed_data = this.m_cos.get_compressed_data();
        trace("Compressed data length (so far) is %d bytes",
                compressed_data.length);
    }

    public void second_write()
    {
        final byte[] additional_data = new byte[] { 0x00, 0x01, 0x02, 0x03,
                0x04, 0x05, 0x06, 0x07, 0x08, 0x09 };
        this.add_data(additional_data);

        final byte[] total_compressed_data = this.m_cos.get_compressed_data();
        trace("Total compressed data length is %d bytes",
                total_compressed_data.length);
    }
}

CompressedOutputStream.java

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.util.zip.Deflater;
import java.util.zip.GZIPOutputStream;

public class CompressedOutputStream extends GZIPOutputStream
{

    private final ByteArrayOutputStream m_out;

    public CompressedOutputStream() throws IOException
    {
        this(32);
    }

    public CompressedOutputStream(int size) throws IOException
    {
        super(new ByteArrayOutputStream(size), size);
        this.m_out = (ByteArrayOutputStream) this.out;
        this.def.setLevel(Deflater.BEST_COMPRESSION);
    }

    public long get_crc32()
    {
        return this.crc.getValue();
    }

    public byte[] get_compressed_data()
    {
        return this.m_out.toByteArray();
    }
}

So I have this code which tries to combine GZIPOutputStream with ByteArrayOutputStream.

For some reason all I'm getting is the same 10 bytes.
Getting this output:

Compressed data length (so far) is 10 bytes
Total compressed data length is 10 bytes
CRC32 of data is 4550d94d

It seems like calling to write() ends-up with an abstractive function, i.e it doesn't get anywhere, nor anything is written.
I want it to be written, compressed, on the fly, and be able to take the compressed bytes at later time.

What am I missing here? Seems trivial but no so.

Edit #1: My final goal
Just to clarify: At the end all I need is an in-memory buffer to which I will write, in chunks and not in a sequence, and at some point when it reaches X bytes I'll be able to take these, compressed & checksum'ed, bytes (to write them somewhere, not a standard Java stream).

Audrius Meškauskas
  • 20,936
  • 12
  • 75
  • 93
Poni
  • 11,061
  • 25
  • 80
  • 121
  • Note: Regarding the "STEPPING INTO THIS IN DEBUG DOESN'T LEAD ANYWHERE" comment, ignore it, it's just that I was using JRE instead of JDK. Now it does enter to the code (step-in)... but the main issue still persists. – Poni Aug 04 '11 at 15:10
  • look at this answer http://stackoverflow.com/q/6717165/779408 – Bob Feb 23 '13 at 10:44

1 Answers1

7

GZIPOutputStream does some internal buffering before it passes the data along to the next stream in the chain (in this case the ByteArrayOutputStream). Thus you have to close() the stream before you can read the bytes back out. For example, I made the following modification to second_write():

this.m_cos.close();
final byte[] total_compressed_data = this.m_cos.get_compressed_data();
trace("Total compressed data length is %d bytes", total_compressed_data.length);

This gave me the output:

Compressed data length (so far) is 10 bytes
Total compressed data length is 98 bytes
CRC32 of data is 4550d94d

If you can't afford to close the stream, there might be a way to forcibly flush the internal buffer without closing. (flush() itself doesn't do that though, I tried).

greghmerrill
  • 695
  • 7
  • 13
  • This might help: http://stackoverflow.com/questions/3640080/force-flush-on-a-gzipoutputstream-in-java – biziclop Aug 04 '11 at 16:15
  • So I have to allocate a NEW object (one of the two) each time?!?!? I can't believe this is all Java has to offer.. – Poni Aug 04 '11 at 16:43
  • @biziclop haven't found anything useful in this page (or simply didn't notice), please point out specifically, if possible. Sorry to bother, and thanks. – Poni Aug 04 '11 at 16:44
  • @Poni - is constructing a new object a bad thing? Imagine for a moment that instead of extending GZIPOutputStream, CompressedOutputStream *contains* GZIPOutputStream and delegates all calls internally to that object. But it keeps track of the number of bytes passed. When that number reaches a certain threshold, you close the stream (and write them to the aforementioned "somewhere"), then create a new GZIPOutputStream for the next batch of calls. – greghmerrill Aug 04 '11 at 17:05
  • First thanks greghmerrill, forgot that. Now, I could do that, maybe, BUT I write all the compressed data to somewhere (look at it as a file), and I was hoping it will be one continuous compressed file, and not many smaller compressed chunks. Hope I make sense.. – Poni Aug 04 '11 at 17:19
  • I think I understand your requirement ... but given that the GZip format has a footer (http://en.wikipedia.org/wiki/GZip), I'm not sure you can achieve what you want with GZip? DeflaterOutputStream sounds like a better conceptual fit, but even in this case the java implementation requires you to indicate when you finish() with your input. Something like this open source version of zlib compression might work: http://www.bolet.org/jgz/api/org/bolet/jgz/ZlibOutputStream.html (I've never used it, but it looks like it has support for incremental flushing) – greghmerrill Aug 04 '11 at 18:29