GZIP eats newlines

Question

I have the following code for compressing and decompressing string.

public static byte[] compress(String str)
{
    try
    {
        ByteArrayOutputStream obj = new ByteArrayOutputStream();
        GZIPOutputStream gzip = new GZIPOutputStream(obj);
        gzip.write(str.getBytes("UTF-8"));
        gzip.close();
        return obj.toByteArray();
    }
    catch (IOException e)
    {
        e.printStackTrace();
    }
    return null;
}

public static String decompress(byte[] bytes)
{
    try
    {
        GZIPInputStream gis = new GZIPInputStream(new ByteArrayInputStream(bytes));
        BufferedReader bf = new BufferedReader(new InputStreamReader(gis, "UTF-8"));
        StringBuilder outStr = new StringBuilder();
        String line;
        while ((line = bf.readLine()) != null)
        {
            outStr.append(line);
        }
        return outStr.toString();
    }
    catch (IOException e)
    {
        return e.getMessage();
    }
}

I compress into byte array on windows, and then send the byte array through socket to the linux and uncompress it there. However upon uncompression it seem that all my newline characters are gone.
So I thought that the problem was linux to windows relationship. However I have tried writing a simple program on windows that uses it, and found that the newlines are still gone.
Can anyone shed any light as to what causes it? I can't figure out any explanation.

Lee Meador · Accepted Answer · 2013-10-10T15:54:54.400

I think the problem is here:

while ((line = bf.readLine()) != null)
    {
        outStr.append(line);
    }

The readLine see's the newline char but doesn't include it in the returned value for line

The problem is worse than you think, perhaps.

readLine() gets all the characters up to, but not including, a newline (or some variety of returns and linefeed characters) OR the end of file. So you don't know if the last line you get had a newline on the end or not.

This might not matter, and if so, you can just add this following the other append:

outStr.append('\n');

Some files might end up with an extra line ending at the end of file.

If it does matter, you will need to use read() and then output all the characters you receive. In that case, you might end up with the infamous "What's at the end of the line?" problem you allude to between Windows, Linux and the MacOS and the way they use different combinations of return and new-line characters to end lines.

You were right. `read()` has just failed, but in my case `.append("\n")` works and I am ok if I do happen to get an extra newline, but hasn't happened so far. — Quillion, Oct 10 '13 at 15:57

Stephen C · Answer 2 · 2013-10-10T15:56:49.243

It is not GZIP that is "eating" newlines.

It is this code:

    while ((line = bf.readLine()) != null)
    {
        outStr.append(line);
    }

The readLine() method reads a line (up to and including a line termination sequence) and then returns it without a newline. You then append it to outStr ... without replacing the line termination that was stripped.

But even if you replaced the line termination, you can't guarantee to preserve the actual line termination sequence that was used ... if you do it that way.

I recommend that you replace the readLine() calls with read() calls; i.e. read and then buffer the data one character at a time. It solves two problems at once. It may even be faster, because you are avoiding the unnecessary overhead of assembling line Strings.

I am very sorry, but Lee was first. Additionally I have tried using read() and unfortunately due to different os' using different characters, when I am transferring data from one to another through socket, newline character does get eaten. Thanks for you help though :) the solution was right and I really liked it. — Quillion, Oct 10 '13 at 17:27

GZIP eats newlines

2 Answers2