zlib.error: Error -3 while decompressing: incorrect header check

Question

I have a gzip file and I am trying to read it via Python as below:

import zlib

do = zlib.decompressobj(16+zlib.MAX_WBITS)
fh = open('abc.gz', 'rb')
cdata = fh.read()
fh.close()
data = do.decompress(cdata)

it throws this error:

zlib.error: Error -3 while decompressing: incorrect header check

How can I overcome it?

score 155 · Answer 1 · edited Jun 22 '19 at 20:18

You have this error:

zlib.error: Error -3 while decompressing: incorrect header check

Which is most likely because you are trying to check headers that are not there, e.g. your data follows RFC 1951 (deflate compressed format) rather than RFC 1950 (zlib compressed format) or RFC 1952 (gzip compressed format).

choosing windowBits

But zlib can decompress all those formats:

to (de-)compress deflate format, use wbits = -zlib.MAX_WBITS
to (de-)compress zlib format, use wbits = zlib.MAX_WBITS
to (de-)compress gzip format, use wbits = zlib.MAX_WBITS | 16

See documentation in http://www.zlib.net/manual.html#Advanced (section inflateInit2)

examples

test data:

>>> deflate_compress = zlib.compressobj(9, zlib.DEFLATED, -zlib.MAX_WBITS)
>>> zlib_compress = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS)
>>> gzip_compress = zlib.compressobj(9, zlib.DEFLATED, zlib.MAX_WBITS | 16)
>>> 
>>> text = '''test'''
>>> deflate_data = deflate_compress.compress(text) + deflate_compress.flush()
>>> zlib_data = zlib_compress.compress(text) + zlib_compress.flush()
>>> gzip_data = gzip_compress.compress(text) + gzip_compress.flush()
>>>

obvious test for zlib:

>>> zlib.decompress(zlib_data)
'test'

test for deflate:

>>> zlib.decompress(deflate_data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
zlib.error: Error -3 while decompressing data: incorrect header check
>>> zlib.decompress(deflate_data, -zlib.MAX_WBITS)
'test'

test for gzip:

>>> zlib.decompress(gzip_data)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
zlib.error: Error -3 while decompressing data: incorrect header check
>>> zlib.decompress(gzip_data, zlib.MAX_WBITS|16)
'test'

the data is also compatible with gzip module:

>>> import gzip
>>> import StringIO
>>> fio = StringIO.StringIO(gzip_data)  # io.BytesIO for Python 3
>>> f = gzip.GzipFile(fileobj=fio)
>>> f.read()
'test'
>>> f.close()

automatic header detection (zlib or gzip)

adding 32 to windowBits will trigger header detection

>>> zlib.decompress(gzip_data, zlib.MAX_WBITS|32)
'test'
>>> zlib.decompress(zlib_data, zlib.MAX_WBITS|32)
'test'

using `gzip` instead

or you can ignore zlib and use gzip module directly; but please remember that under the hood, gzip uses zlib.

fh = gzip.open('abc.gz', 'rb')
cdata = fh.read()
fh.close()

@dnozay, I've tried using the above `zlib.decompress(zlib_data, zlib.MAX_WBITS|32)` tweak that you mentioned, but it hasn't worked. I still get the `incorrect header check` error. If I try using the the other options mentioned above, I still get various errors. Is there anything else that could trigger this error? — Minu, Mar 24 '17 at 21:00
@Minu, sure -- any kind of data that's neither valid deflate, zlib or gzip content would fail the header check. — Charles Duffy, Aug 25 '17 at 16:46
`zlib.MAX_WBITS | 16` worked for me, thanks. It is amazingly non-trivial to deduce this from [the documentation](https://docs.python.org/3.6/library/zlib.html#zlib.decompress). Also, it's annoying that `aiohttp` doesn't decode `Content-Encoding: gzip` transparently. — user4815162342, Dec 10 '17 at 19:03

score 4 · Accepted Answer · edited May 23 '17 at 12:10

4

Update: dnozay's answer explains the problem and should be the accepted answer.

Try the gzip module, code below is straight from the python docs.

import gzip
f = gzip.open('/home/joe/file.txt.gz', 'rb')
file_content = f.read()
f.close()

edited May 23 '17 at 12:10

Community

1
1

answered Jun 26 '10 at 00:13

Dave Bacher

15,652
3
63
86

Same error presented : Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.6/gzip.py", line 212, in read self._read(readsize) File "/usr/lib/python2.6/gzip.py", line 271, in _read uncompress = self.decompress.decompress(buf) zlib.error: Error -3 while decompressing: invalid code lengths set – VarunVyas Jun 26 '10 at 00:18
@VarunVyas, sorry, I can't reproduce your error. It must have something to do with your input data. Was your input file generated with gzip? Does gunzip from the command line decompress it correctly? – Dave Bacher Jun 26 '10 at 04:09

score 3 · Answer 3 · answered Jun 11 '13 at 16:08

I just solved the "incorrect header check" problem when uncompressing gzipped data.

You need to set -WindowBits => WANT_GZIP in your call to inflateInit2 (use the 2 version)

Yes, this can be very frustrating. A typically shallow reading of the documentation presents Zlib as an API to Gzip compression, but by default (not using the gz* methods) it does not create or uncompress the Gzip format. You have to send this non-very-prominently documented flag.

djvg · Answer 4 · 2021-08-19T09:12:55.510

This does not answer the original question, but it may help someone else that ends up here.

The zlib.error: Error -3 while decompressing: incorrect header check also occurs in the example below:

b64_encoded_bytes = base64.b64encode(zlib.compress(b'abcde'))
encoded_bytes_representation = str(b64_encoded_bytes)  # this the cause
zlib.decompress(base64.b64decode(encoded_bytes_representation))

The example is a minimal reproduction of something I encountered in some legacy Django code, where Base64 encoded bytes (from an HTTP POST) were being stored in a Django CharField (instead of a BinaryField).

When reading a CharField value from the database, str() is called on the value, without an explicit encoding, as can be seen in the Django source.

The str() documentation says:

If neither encoding nor errors is given, str(object) returns object.str(), which is the “informal” or nicely printable string representation of object. For string objects, this is the string itself. If object does not have a str() method, then str() falls back to returning repr(object).

So, in the example, we are inadvertently base64-decoding

"b'eJxLTEpOSQUABcgB8A=='"

instead of

b'eJxLTEpOSQUABcgB8A=='.

The zlib decompression in the example would succeed if an explicit encoding were used, e.g. str(b64_encoded_bytes, 'utf-8').

NOTE specific to Django:

What's especially tricky: this issue only arises when retrieving a value from the database. See for example the test below, which passes (in Django 3.0.3):

class MyModelTests(TestCase):
    def test_bytes(self):
        my_model = MyModel.objects.create(data=b'abcde')
        self.assertIsInstance(my_model.data, bytes)  # issue does not arise
        my_model.refresh_from_db()
        self.assertIsInstance(my_model.data, str)  # issue does arise

where MyModel is

class MyModel(models.Model):
    data = models.CharField(max_length=100)

Asclepius · Answer 5 · 2019-06-24T11:03:05.237

To decompress incomplete gzipped bytes that are in memory, the answer by dnozay is useful but it misses the zlib.decompressobj call which I found to be necessary:

incomplete_decompressed_content = zlib.decompressobj(wbits=zlib.MAX_WBITS | 16).decompress(incomplete_gzipped_content)

Note that zlib.MAX_WBITS | 16 is 15 | 16 which is 31. For some background about wbits, see zlib.decompress.

Credit: answer by Yann Vernier which notes the the zlib.decompressobj call.

Yury Bondarau · Answer 6 · 2019-06-07T08:04:25.763

My case was to decompress email messages that are stored in Bullhorn database. The snippet is the following:

import pyodbc
import zlib

cn = pyodbc.connect('connection string')
cursor = cn.cursor()
cursor.execute('SELECT TOP(1) userMessageID, commentsCompressed FROM BULLHORN1.BH_UserMessage WHERE DATALENGTH(commentsCompressed) > 0 ')



 for msg in cursor.fetchall():
    #magic in the second parameter, use negative value for deflate format
    decompressedMessageBody = zlib.decompress(bytes(msg.commentsCompressed), -zlib.MAX_WBITS)

score 1 · Answer 7 · answered Nov 18 '10 at 20:38

Funnily enough, I had that error when trying to work with the Stack Overflow API using Python.

I managed to get it working with the GzipFile object from the gzip directory, roughly like this:

import gzip

gzip_file = gzip.GzipFile(fileobj=open('abc.gz', 'rb'))

file_contents = gzip_file.read()

score 0 · Answer 8 · edited Aug 03 '23 at 01:50

If you use Node.js , try fflate package, that worked for me for gzip.

const fflate = require('fflate');


    const decompressedData = await new Promise((resolve, reject) => {
           fflate.gunzip(buffer, (error, result) => {
                       if (error) {
                       reject(error);
                       } else {
                       resolve(result);
                      }
                   });
                });
           xml = Buffer.from(decompressedData).toString('UTF-8');

score -3 · Answer 9 · answered Dec 08 '17 at 08:19

-3

Just add headers 'Accept-Encoding': 'identity'

import requests

requests.get('http://gett.bike/', headers={'Accept-Encoding': 'identity'})

https://github.com/requests/requests/issues/3849

answered Dec 08 '17 at 08:19

Barny

383
1
3
13

4

So your answer to a decompression question is: don't compress it to start with?? – Ryder Brooks Dec 14 '17 at 20:32
1

Servers don't always respect the stated header, so this doesn't reliably work. – Asclepius Jun 22 '19 at 20:11

zlib.error: Error -3 while decompressing: incorrect header check

9 Answers9

choosing windowBits

examples

automatic header detection (zlib or gzip)

using `gzip` instead

Linked

zlib.error: Error -3 while decompressing: incorrect header check

9 Answers9

choosing windowBits

examples

automatic header detection (zlib or gzip)

using gzip instead

Linked

using `gzip` instead