Different behaviours for gz files decompress process

Question

I have several gz files on which decompression seems to not be properly working. I have selected one of those, this file is in a local folder and it came from an external source. I don't know the compression process.

I have created a python script to illustrate the situation. In order to have a reference file for testing I have uncompressed the gz file and compressed it again (using gzip on Ubuntu) to generate the same gz file in my computer. Those two files present different behaviors:

    import gzip
    import zlib
    import hashlib

    def md5(content):
        m = hashlib.md5()
        m.update(content)
        return m.hexdigest()

    def decompress_gzip_size(file_name):
        with gzip.open(file_name, 'rb') as f_out:
            f_content = f_out.read()
            print(len(f_content), file_name)
            print (md5(f_content), file_name)

    def decompress_open_gzip_size(file_name):
        with open(file_name, 'rb') as f_out:
            f_content = f_out.read()
            unzip_content = gzip.decompress(f_content)
            print(len(unzip_content), file_name)
            print (md5(unzip_content), file_name)

    def decompress_zlib_size(file_name):
        with open(file_name, 'rb') as f_out:
            f_content = f_out.read()
            unzip_content = zlib.decompress(f_content, 32)
            print(len(unzip_content), file_name)
            print (md5(unzip_content), file_name)

    def decompress_zlib_obj(file_name):
        decompress_obj = zlib.decompressobj(32)
        with open(file_name, 'rb') as f_out:
            f_content = f_out.read()
            unzip_content = decompress_obj.decompress(f_content)
            print(len(unzip_content), file_name)
            print(len(decompress_obj.unused_data), 'Unused data')
            print (md5(unzip_content), file_name)

    external_file = 'external_source_compress.gz'
    my_file = 'my-compress-file.gz'

    print("decompress_gzip_size")
    decompress_gzip_size(my_file)
    decompress_gzip_size(external_file)
    print("*" * 60)

    print("decompress_open_gzip_size")
    decompress_open_gzip_size(my_file)
    decompress_open_gzip_size(external_file)
    print("*" * 60)

    print("decompress_zlib_size")
    decompress_zlib_size(my_file)
    decompress_zlib_size(external_file)
    print("*" * 60)

    print("decompress_zlib_obj")
    decompress_zlib_obj(my_file)
    decompress_zlib_obj(external_file)
    print("*" * 60)

Execution output is:

    decompress_gzip_size
    167019534 my-compress-file.gz
    a4dd17dd28b89f0b2c300b607cd1a8ba my-compress-file.gz
    167019534 external_source_compress.gz
    a4dd17dd28b89f0b2c300b607cd1a8ba external_source_compress.gz
    ************************************************************
    decompress_open_gzip_size
    167019534 my-compress-file.gz
    a4dd17dd28b89f0b2c300b607cd1a8ba my-compress-file.gz
    167019534 external_source_compress.gz
    a4dd17dd28b89f0b2c300b607cd1a8ba external_source_compress.gz
    ************************************************************
    decompress_zlib_size
    167019534 my-compress-file.gz
    a4dd17dd28b89f0b2c300b607cd1a8ba my-compress-file.gz
    33408639 external_source_compress.gz
    4f51ccc64a7baab5ee5e2ce31e816409 external_source_compress.gz
    ### SIZES AND MD5 DO NOT MATCH ###
    ************************************************************
    decompress_zlib_obj
    167019534 my-compress-file.gz
    0 Unused data
    a4dd17dd28b89f0b2c300b607cd1a8ba my-compress-file.gz
    33408639 external_source_compress.gz
    46765202 Unused data
    4f51ccc64a7baab5ee5e2ce31e816409 external_source_compress.gz
    ### THERE IS SOME UNUSED DATA IN THE ORIGINAL FILE ###
    ************************************************************

NOTE: zlib decompression fails if window size parameters is any other than 32.

Without setting the window-size this happend:

    Traceback (most recent call last):
      File "decompress_python.py", line 53, in <module>
        decompress_zlib_size(my_file)
      File "decompress_python.py", line 26, in decompress_zlib_size
        unzip_content = zlib.decompress(f_content)
    zlib.error: Error -3 while decompressing data: incorrect header check

As it can be seen. Without 32 windows-size, the script crash, but with window-size 32 it finish, but the amount of read bytes is just a portion of the real amount of data.

File has only one element inside:

    gzip -l external_source_compress.gz
             compressed        uncompressed  ratio uncompressed_name
               58609586            33410520 -75.4% external_source_compress

Could anyone help me to understand what is happening here? I'm truly lost. Thanks in advance.

The fact that the `gzip -l` reported compressed size is so much larger than the uncompressed size proves that it must have multiple members. gzip only expands incompressible data by a fraction of a percent, so it is not possible for the compressed data to be that much larger than the uncompressed data for a single member. — Mark Adler, Oct 04 '18 at 21:08

Mark Adler · Accepted Answer · 2018-10-06T00:17:49.857

2

Florian's answer is almost certainly what's happening, which is the gzip file you are only partially decompressing must have multiple members. To decompress the next member, you can simply get the unused_data from the decompressobj object and start another decompression. Repeat until all of the input is used.

This answer shows an example of processing a gzip file with multiple members in python.

It sounds like what you should be using for the wbits parameter is 31, not 32. 31 expects a gzip wrapper as opposed to a zlib wrapper. Not specifying wbits results in the decompressor expecting a zlib wrapper, which is why it doesn't work on your gzip streams.

edited Oct 06 '18 at 00:17

answered Oct 04 '18 at 02:05

Mark Adler

101,978
13
118
158

Thanks Mark. I'm actually not looking for a solution in Python. I'm trying to get a way around this behavior for some gzip in Java/Scala, I just want to understand why this happen. Python script is just the way to explain the situation. Why wbits defines the format of the file? That's strange to me. – pez_betta Oct 04 '18 at 09:49
I have added a `gzip -l` output of the file. As you can see there there is only one file within it. Also the negative compression ratio makes think something is wrong with the file. What do you think? – pez_betta Oct 04 '18 at 10:02
2

`gzip -l` does not correctly handle multiple gzip members. It tells you nothing. Though the negative compression level is at least a clue that there are multiple members. – Mark Adler Oct 04 '18 at 20:32
1

If you have pigz, you can use `pigz -ltv`, which does handle multiple gzip members. – Mark Adler Oct 04 '18 at 20:33
But what does it means that have multiple members? There is only one file to extract. Gzip or any other compression (I have tried 4, in Win 10 or Linux) tool just give me one file. – pez_betta Oct 05 '18 at 07:15
Yes, there is one file encoded as multiple members. – Mark Adler Oct 05 '18 at 15:38
1

@pez_betta I have added a link to another answer here with an example, in Python, of how to do this. – Mark Adler Oct 06 '18 at 00:18

score 1 · Answer 2 · answered Oct 03 '18 at 10:36

gzip streams can consist of multiple items compressed independently, which are then simply concatenated. The expectation is that a gzip-style decompressor transparently reads through all independently compressed parts, producing a single output stream (basically ignoring the internal end-of-stream indicators).

The documentation of zlib.compressobj does not say it, but it will stop at the first zlib-detected internal end of stream condition. The remaining data could then be processed independently. In some applications (but not yours), it may not even be zlib-compressed data.

Thanks for your answer, that's actually helpful. Could you try to explain me the window-size error? I have edited the question to add more detail. — pez_betta, Oct 03 '18 at 11:16

Different behaviours for gz files decompress process

2 Answers2