How can I gzip an InputStream and return an InputStream?

Question

I am starting with a response from a HTTP request:

InputStream responseInputStream = response.getEntityInputStream()

I need to gzip that response so I can upload it to s3 and save it compressed:

this.s3.putObject(new PutObjectRequest(bucketName, key, gzippedResponseInputStream, meta));

I am aware that I can get the byte[] array out of responseInputStream and then gzip them into a new InputStream. However, that can be very inefficient with a large amount of data.

I know that there have been similar questions asked on SO, but I have not found anything that seems to address the specific need of starting with an InputStream and finishing with a gzipped InputStream.

Thanks for any help!

Have you checked this answer ? http://stackoverflow.com/a/19326492/2588800 — Svetlin Zarev, Aug 12 '16 at 19:39
Wow, thank you @SvetlinZarev - that worked! I have been searching through SO trying every "solution" that I have found and none of that have worked so far, but your suggestion did the trick! Thank you! — chaimp, Aug 12 '16 at 20:37

Svetlin Zarev · Answer 1 · 2016-08-12T20:12:17.823

public final class Example {
    public static void main(String[] args) throws IOException, InterruptedException {
        final PipedInputStream inputStream = new PipedInputStream();
        final PipedOutputStream outputStream = new PipedOutputStream(inputStream);

        Thread compressorThread = new Thread() {
            @Override
            public void run() {
                try (FileInputStream dataSource = new FileInputStream(args[0])) {
                    try (GZIPOutputStream sink = new GZIPOutputStream(outputStream)) {
                        final byte[] buffer = new byte[8 * 1024];
                        for (int bytesRead = dataSource.read(buffer); bytesRead >= 0; bytesRead = dataSource.read(buffer)) {
                            sink.write(buffer, 0, bytesRead);
                        }
                    }
                } catch (IOException ex) {
                    //TODO handle exception -> maybe use callable + executor
                }
            }
        };
        compressorThread.start();

        try (FileOutputStream destination = new FileOutputStream(args[1])) {
            final byte[] buffer = new byte[8 * 1024];
            for (int bytesRead = inputStream.read(buffer); bytesRead >= 0; bytesRead = inputStream.read(buffer)) {
                destination.write(buffer, 0, bytesRead);
            }
        }

        compressorThread.join();
    }

}

You are right, my previous example was wrong. You can use piped streams. The catch here is that you cannot use the input and output stream from the same thread. Also don't forget to join() on the writing thread. You can test my example by supplyng two parameters:

args[0] -> the source file
args[1] -> the destination to write the compressed content

PS: @11thdimension was a few minutes faster with his piped stream solutions, so if you find this helpful please accept his answer

The GZIPInputStream is for taking already compressed data and decompressing it. I actually tried this already before realizing that. But, it says so clearly in the documentation as well. Thank you for the fast response, but this is not correct. — chaimp, Aug 12 '16 at 19:13
Thank you @svetlin I ended going with the solution that you found in the comment to the question, but I appreciate this and I trust that it works, so I will mark it as "accepted". — chaimp, Aug 12 '16 at 20:41

11thdimension · Answer 2 · 2016-08-12T20:11:44.427

3

I think you're looking for a PipedInputStream

Here's how it can be done.

public InputStrema getGZipStream() {
    final PipedOutputStream pos = new PipedOutputStream();
    PipedInputStream pis = new PipedInputStream();

    try (final InputStream responseInputStream = response.getEntityInputStream();
    ){
        pis.connect(pos);

        Thread thread = new Thread() {
            public void run () {
                startWriting(pos, responseInputStream);
            }
        };
        thread.start();
    } catch(Exception e) {
        e.printStackTrace();
    }

    return pis;
}

public void startWriting(OutputStream out, InputStream in) {
    try (GZIPOutputStream gOut = GZIPOutputStream(out);) {
        byte[] buffer = new byte[10240];
        int len = -1;
        while ((len = in.read(buffer)) != -1) {
            gOut.write(buffer, 0, len);
        }
    } catch (Exception ex) {
        ex.printStackTrace();
    } finally {
        try {
            out.close();
        } catch( Exception e) {
            e.printStackTrace();
        }
    }
}

I haven't tested this code, please let me know if this works.

edited Aug 12 '16 at 20:11

answered Aug 12 '16 at 19:49

11thdimension

10,333
4
33
71

1

It won;t work because you'll close the `pis` when you exit the `try` body - so you'll return a closed stream; – Svetlin Zarev Aug 12 '16 at 20:03
I was thinking the same thing. Does the safe Try close streams at the end of the method ? Anyway I will modify the code to work with it. – 11thdimension Aug 12 '16 at 20:09
Changed, it should not have closed stream problem now at least. – 11thdimension Aug 12 '16 at 20:12
Thanks, I will give this a try if the other solution(s) give any difficulty. I appreciate the fast response, and I am sure this will be useful for others too. – chaimp Aug 12 '16 at 20:42

How can I gzip an InputStream and return an InputStream?

2 Answers2