3

I am trying to sift through a big database that is compressed in a .zst. I am aware that I can simply just decompress it and then work on the resulting file, but that uses up a lot of space on my ssd and takes 2+ hours so I would like to avoid that if possible.

Often when I work with large files I would stream it line by line with code like

with open(filename) as f:
    for line in f.readlines():
        do_something(line)

I know gzip has this

with gzip.open(filename,'rt') as f:
    for line in f:
        do_something(line)

but it doesn't seem to work with .zsf, so I am wondering if there're any libraries that can decompress and stream the decompressed data in a similar way. For example:

with zstlib.open(filename) as f:
    for line in f.zstreadlines():
        do_something(line)
SimonUnderwood
  • 469
  • 3
  • 12

0 Answers0