Is GZIPOutputStream known to loose data during compression?

Question

I have a very strange issue with GZIPOutputStream when compressing an array of doubles. At the 57th element, I get a small discrepancy when I reload the data:

57 > 3.003727492141554E7 3.0037273913440887E7
1900: false
57 > -6.110783629228158E7 -6.110783629076558E7
2000: false

1900 and 2000 are two different set of doubles. The left value is the original value.

When I use a simple FileOutputStream without GZIP, I don't get the issue. Why? Is GZIP output stream known to loose information?

EDIT

Here is how I read and write the data:

public static final double[] coefficients = new double[1161289];

...

public static void dump(File f) throws FileNotFoundException, IOException {

    OutputStream os = FileUtils.zipContent(f);

    byte[] ba = new byte[8];
    ByteBuffer BF = ByteBuffer.wrap(ba);
    BF.order(ByteOrder.BIG_ENDIAN);

    for (int i=0;i<coefficients.length;i++) {
        BF.putDouble(0, coefficients[i]);
        os.write(ba,0,8);
    }

    os.close();

}

public static void load(File f) throws FileNotFoundException, IOException {

    InputStream is = FileUtils.readZippedContent(f);

    byte[] ba = new byte[8];
    final ByteBuffer BF = ByteBuffer.wrap(ba);
    BF.order(ByteOrder.BIG_ENDIAN);

    for (int i=0;i<coefficients.length;i++) {
        is.read(ba,0,8);
        coefficients[i] = BF.getDouble(0);
    }

}

...

public static GZIPOutputStream zipContent(File f)
        throws FileNotFoundException, IOException {

    return new GZIPOutputStream(new FileOutputStream(f));

}

public static GZIPInputStream readZippedContent(File f)
        throws FileNotFoundException, IOException {

    return new GZIPInputStream(new FileInputStream(f));

}

How, **exactly,** are you writing the array of doubles to the output stream? — Matt Ball, Jul 16 '11 at 19:50
That is NOT a SSCCE!!! Where do you invoke the load/dumpe methods? Where do you add data to the array? Where is the FileUtils class? — camickr, Jul 16 '11 at 20:17
@leonbloy I tried using DataOutputStream and the issue is solved. Create an answer and I'll approve it. — Jérôme Verstrynge, Jul 16 '11 at 20:22
look at this answer http://stackoverflow.com/q/6717165/779408 — Bob, Feb 23 '13 at 10:45

score 4 · Answer 1 · answered Jul 16 '11 at 20:58

4

what makes you think that: is.read(ba,0,8); returns 8 always?

In short: the reading routine is bogus.

answered Jul 16 '11 at 20:58

bestsss

11,796
3
53
63

Because what I put in this file is a multiple of 8 – Jérôme Verstrynge Jul 17 '11 at 06:00
@JVerstry, you'd be better off reading the doc of InputStream#read... and then I know for a fact zip streams may not return full amount of data when there are leftover bytes in their internal window. – bestsss Jul 17 '11 at 07:41
This could explain my issue indeed. – Jérôme Verstrynge Jul 17 '11 at 07:47
@JVerstry, yes, always, i mean always check the returned value of `read(...)` function. You are dealing w/ IO, the stream can be corrupter/cut, so you must be prepared handling errors. – bestsss Jul 17 '11 at 07:54

jcomeau_ictx · Accepted Answer · 2011-07-16T19:57:59.937

1

gzip is lossless compression. any missing data would have to be accounted for by some other problem.

if the 57th element is the final element, I'd suspect failure to close the output file before re-opening for reading.

edited Jul 16 '11 at 19:57

answered Jul 16 '11 at 19:52

jcomeau_ictx

37,688
6
92
107

57th element is not the final element. – Jérôme Verstrynge Jul 16 '11 at 20:01
I have changed my routine to use DataOutputStream instead and the issue is solved. – Jérôme Verstrynge Jul 17 '11 at 06:47

Is GZIPOutputStream known to loose data during compression?

2 Answers2