How to determine the Content-Length of a gzipped file in Python?

Question

I've a big compressed file and I want to know the size of the content without uncompress it. I've tried this:

import gzip
import os

with gzip.open(data_file) as f:
          f.seek(0, os.SEEK_END)
          size = f.tell()

but I get this error

ValueError: Seek from end not supported

How can I do that?

Thx.

The manual says that [`os.SEEK_END`](https://docs.python.org/2/library/os.html#os.SEEK_END) is for [`lseek`](https://docs.python.org/2/library/os.html#os.lseek). And this is probably what you need- — llrs, Jun 20 '14 at 17:15
possible duplicate of [Get uncompressed size of a .gz file in python](http://stackoverflow.com/questions/1704458/get-uncompressed-size-of-a-gz-file-in-python) — Christian Berendt, Jun 20 '14 at 17:27
@Llopis, according to [the `file` docs for Python 2.x](//docs.python.org/2/library/stdtypes.html#file.seek), the `os.SEEK_` constants are indeed the right ones to use with `file.seek`. Also, `os.lseek` won't work here because the `gzip` file-like object does not have an underlying POSIX-level file descriptor. — Dan Lenski, Jun 20 '14 at 17:29

score 2 · Answer 1 · edited Dec 12 '17 at 21:18

2

Unfortunately, the Python 2.x gzip module doesn't appear to support any way of determining uncompressed file size.

However, gzip does store the uncompressed file size as a little-endian 32-bit unsigned integer at the very end of the file: http://www.abeel.be/content/determine-uncompressed-size-gzip-file

Unfortunately, this only works for files <4gb in size due to using only a 32-bit integer the gzip format; see the manual.

import os
import struct

with open(data_file,"rb") as f:
    f.seek(-4, os.SEEK_END)
    size, = struct.unpack("<I", f.read(4))
    print size

edited Dec 12 '17 at 21:18

Mathieu IKO

55
4

answered Jun 20 '14 at 17:24

Dan Lenski

76,929
13
76
124

This approach has other issues besides the 4 GB limit. – Mark Adler Jun 21 '14 at 13:24
Other than the fact that not every kind of file is `seek`-able, what would those be? – Dan Lenski Jun 21 '14 at 13:28
1

See my answer, including link to another answer. – Mark Adler Jun 21 '14 at 14:53
The problem is that I'm working on archive of about 10 GB so your technique wont work here. Thank you anyway! – Jprog Jun 23 '14 at 16:56

score 2 · Answer 2 · edited May 23 '17 at 12:24

It is not possible in principle to definitively determine the size of the uncompressed data in a gzip file without decompressing it. You do not need to have the space to store the uncompressed data -- you can discard it as you go along. But you have to decompress it all.

If you control the source of the gzip file and can assure that a) there are no concatenated members in the gzip file, b) the uncompressed data is less than 4 GB in length, and c) there is no extraneous junk at the end of the gzip file, then and only then you can read the last four bytes of the gzip file to get a little-endian integer that has the length of the uncompressed data.

See this answer for more details.

Here is Python code to read a gzip file and print the uncompressed length, without having to store or save the uncompressed data. It limits the memory usage to small buffers. This requires Python 3.3 or greater:

#!/usr/local/bin/python3.4
import sys
import zlib
import warnings
f = open(sys.argv[1], "rb")
total = 0
buf = f.read(1024)
while True:             # loop through concatenated gzip streams
    z = zlib.decompressobj(15+16)
    while True:         # loop through one gzip stream
        while True:     # go through all output from one input buffer
            total += len(z.decompress(buf, 4096))
            buf = z.unconsumed_tail
            if buf == b"":
                break
        if z.eof:
            break       # end of a gzip stream found
        buf = f.read(1024)
        if buf == b"":
            warnings.warn("incomplete gzip stream")
            break
    buf = z.unused_data
    z = None
    if buf == b"":
        buf == f.read(1024)
        if buf == b"":
            break
print(total)

Thanks for your answer, but I'm not sure of what your code does: Does it gives the number of elements in the data file or the size (in bytes) of the uncompressed file? And which function in your code needs python3.3 ? Because I'm running on machine on which I don't have the choice of the version of python (2.7 probably) — Jprog, Jun 23 '14 at 16:57
The uncompressed size in bytes. The `eof` method for `decompressobj` was added in 3.3, and is needed to recognize when you have reached the end of a gzip stream. — Mark Adler, Jun 23 '14 at 18:02

score -2 · Accepted Answer · answered Jun 26 '14 at 10:56

-2

To summerize, I need to open huges compressed files (> 4GB) so the technique of Dan won't work and I want the length (number of line) of the file so the technique of Mark Adler is not appropriate.

Eventually, I found for uncompressed files a solution( not the most optimized but it works!) which can be transposed easily to compressed files:

size = 0

with gzip.open(data_file) as f:
  for line in f:
    size+= 1
    pass

return size

Thank you all, people in this forum are very effective!

answered Jun 26 '14 at 10:56

Jprog

79
1
8

2

Your question did not ask for the number of lines. In fact your example of an attempt, if it had worked, would return the number of bytes from `f.tell()`, not the number of lines. – Mark Adler Aug 21 '15 at 15:29

How to determine the Content-Length of a gzipped file in Python?

3 Answers3

Linked