open large gzip file (~1gb) in python

Question

I am beginner in python and trying to learn python. I have written few line of code to open a large gzip file (size of ~ 1gb) and want to extract some content, however I am getting memory related error. could somebody please guide me how open the gzip with limited memory. I have put a part of code that is throwing error.

import os
import gzip

with gzip.open("test.gz","rb") as peak:
     for line in peak:
         file_content = line.read().decode("utf-8")             
         print(file_content)

Error: File "/software/anaconda3/lib/python3.7/gzip.py", line 276, in read return self._buffer.read(size)

Sure you're not hitting the memory limit? Consider reading the file in chunks. You can iterate over the fileObject returned from `gzip.open`. — Eliran Abdoo, Aug 09 '19 at 13:51
Thanks @EliranAbdoo for reply ! I tried to iterate over the object but its throwing another error ` file_content = line.read() AttributeError: 'bytes' object has no attribute 'read' ` — Gyan Prakash Mishra, Aug 09 '19 at 14:06
@GyanPrakashMishra it's because the line is the context itself. As the interpreter says, it's a bytes object (so just remove the `read()` and decode it directly. — Eliran Abdoo, Aug 09 '19 at 14:08

score 1 · Answer 1 · answered Aug 09 '19 at 13:48

I am trying to recreate your issue but I am unable to. Using fallocate I create a big file, then gzip it, but hit no error in Python

$ fallocate -l 2G tempfile.img
$ gzip tempfile.img
$ ipython
>>> import gzip
>>> with gzip.open('tempfile.img.gz', 'rb') as fIn:
>>>    content = fIn.read()

If you hit an exception, it should have some name like OSError or something more specific. My guess is that you have a 32-bit installation of Python which would impose memory limits in the gigabyte range. This SO thread covers a way to check if you're running 32- or 64-bit.

If you post the name of the exception or a reproducible example, then I can update this answer.

Here is architecture of system and installed 64-bit only `>>> import platform >>> platform.architecture() ('64bit', '') >>> ` ok I will try to reproduce the example — Gyan Prakash Mishra, Aug 09 '19 at 13:55

open large gzip file (~1gb) in python

1 Answers1