0

I have been trying to read 3GB of gzip file. I have extracted it with gzip but after extracting it cannot even fit in 60GB of storage. So If I can't extract which is JSON so I can't read in bytes. I have found many questions all those were downloaded then extract into bytes and then read in bytes but here I can't even extract them.

So if it is possible to read while downloading in bytes? I have tried but the error says need to download the whole file.

Downloading Code :

default_path_download = '.'
def saveanddelete(download_url, name):
  with requests.get(download_url, headers=headersfordowlnoading, stream=True) as r:
      content_lenght = int(r.headers['Content-Length'])
      print(f"Download File Size : {round(content_lenght/1000000, 5)} MB")
      r.raise_for_status()
      with open(f"{default_path_download}/{name}", 'wb') as f:
          total_chunk_downloaded = 0
          chunk_size_to_get = 4096
          for chunk in r.iter_content(chunk_size=chunk_size_to_get):
              total_chunk_downloaded += len(chunk)
              if chunk:
                  f.write(chunk)
                  done = int(50 * total_chunk_downloaded / content_lenght)
                  print(f"Downloaded :  ", end='',flush=True)
                  print(f"{round(total_chunk_downloaded/1000000, 5)} MB [{'=' * done}{' ' * (50-done)}]\r", end='', flush=True)
                  break
          print("\n")
          print("Download Complete")

You can see the whole code on colab I would like to other methods also if possible? Thanks For Your Help!

Adarsh Raj
  • 325
  • 4
  • 17
  • You can use the pipe operator (`|`) to directly feed the decompressed output into the script (something like `gunzip -c archive.gz | python script.py`). There are streaming JSON decoder modules for python (for example, [`json-stream`](https://pypi.org/project/json-stream/)). – Ouroborus Jul 13 '22 at 07:17
  • Also, python comes with a streaming [gzip module](https://docs.python.org/3/library/gzip.html) – Ouroborus Jul 13 '22 at 07:30
  • I have no idea to elevate it. – Adarsh Raj Jul 13 '22 at 13:54

0 Answers0