2

I am using the zstandard library to pack and unpack. I am essentially trying to recreate the following bash in Python:

curl -O - http://download.com/file.tar.zst | zstd -d - | tar xf -

I found this discussion saying that Python's tarfile module supports any stream so using zstd should be trivial. I am having trouble using the streams, though.

I tried to do the following:

import requests
import zstandard
import tarfile

# Write .tar.zst archive
path = "/some/path/to/tar"
to_zstd = zstd.ZstdCompressor()
with tempfile.NamedTemporaryFile() as temp_tar_file:
  with tarfile.open(name="file.tar.zst", fileobj=to_zstd.stream_writer(temp_tar_file), mode="w|") as tf:
    tf.add(path)

# Upload generated .tar.zst

# Extract .tar.zst
with closing(requests.get(url, stream=True) as r:
  tardir = tempfile.mkdtemp()
  unzstd = zstd.ZstdDecompressor()
  with io.BytesIO() as tarbyte:
    unzstd.copy_stream(resp.raw, tarbyte)
    with tarfile.open(fileobj=tarbyte, mode="r|") as tf:
      tf.extractall(path=tardir)

Running this I get tarfile.ReadError("empty file") and running zstd -d file.tar.zst I get Read error (39) : premature end. The file size of file.tar.zst isn't zero and it does change depending on what's compressed so it's not empty, but I'm still not compressing right. Where am I going wrong?

drowsily.throbbing872
  • 21
  • 2

1 Answers1

0

I got this working, don't know if it's the best way.

# Compression
with open(tarpath, 'wb') as to_stream:
  to_zstd = zstandard.ZstdCompressor()
  with to_zstd.stream_writer(to_stream) as stream:
    with tarfile.open(mode="w|", fileobj=stream) as tf:
      tf.add(path)

# Inflation
with closing(requests.get(url, stream=True) as r:
  unzstd = zstandard.ZstdDecompressor()
  with unzstd.stream_reader(r.raw) as stream:
    with tarfile.open(mode="r|", fileobj=stream) as tf:
      tf.extractall(path)
drowsily.throbbing872
  • 21
  • 2
  • Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jul 17 '23 at 20:22