Stack Overflow
Stack Overflow

Questions tagged [zstandard]

Real-time compression algorithm and software.

Zstandard is a real-time compression algorithm, providing high compression ratios. It offers a very wide range of compression / speed trade-off, while being backed by a very fast decoder (see benchmarks below). It also offers a special mode for small data, called dictionary compression, and can create dictionaries from any sample set. Zstandard library is provided as open source software using a BSD license.

32 questions
9
votes
1 answer

How to decompress lzma2 (.xz) and zstd (.zst) files into a folder using Python 3?

I have been working for a long time with .bz2 files. To unpack/decompress .bz2 files into a specific folder I have been using the following function: destination_folder = 'unpacked/' def decompress_bz2_to_folder(input_file): unpackedfile =…
Aventinus
  • 1,322
  • 2
  • 15
  • 33
8
votes
1 answer

Read Zstandard-compressed file in Spark 2.3.0

Apache Spark supposedly supports Facebook's Zstandard compression algorithm as of Spark 2.3.0 (https://issues.apache.org/jira/browse/SPARK-19112), but I am unable to actually read a Zstandard-compressed file: $ spark-shell ... // Short name throws…
Josh Johnson
  • 8,832
  • 4
  • 25
  • 31
7
votes
1 answer

zstd: error 70 : Write error : Broken pipe (cannot write decoded block)

I'm trying through Windows terminal to decompress a large number of compressed files with zstd v1.4.0 and then 'ag' search over : zstd -dc -r . | ag -z -i "term" It gives me the following error while proceeding : zstd: error 70 : Write error :…
titibouboul
  • 1,348
  • 3
  • 16
  • 30
5
votes
1 answer

Warnings while building Rakudo in Windows

I followed the instructions for building Rakudo here. With similar reading here and here, I tried building it in Windows with VS-2019. However, while building Rakudo in windows I get the following build warnings: Updating submodules…
Suman Khanal
  • 3,079
  • 1
  • 17
  • 29
5
votes
1 answer

Decompressing large zst files

I have a 16gb compressed zst file. how would I decompress it on windows? I do not have the memory to handle the decompressed version, so it needs to be written straight to a file.
RRKS101_1 TF2
  • 89
  • 1
  • 1
  • 7
5
votes
3 answers

How to compress parquet file with zstandard using pandas

i'm using pandas to convert dataframes to .parquet files using this command: df.to_parquet(file_name, engine='pyarrow', compression='gzip') I need to use zstandard as compression algorithm, but the function above accepts only gzip, snappy, and…
alcor
  • 515
  • 1
  • 8
  • 21
4
votes
1 answer

Using zstandard to compress a file in Python

So I'm using the zstandard python library, and I've written a helper class and function to use contexts to decompress files. class ZstdReader: def __init__(self, filename): self.filename = filename def __enter__(self): …
John Do
  • 186
  • 12
4
votes
2 answers

Make zstd compressed files 'rsyncable' like gzip does with --rsyncable option

Is there a way to make zstd compressed files 'rsyncable' like gzip does with --rsyncable option? I've tried splitting input files into fixed length chunks and compressing them separately with no luck. About the --rsyncable option: When you…
silinxey
  • 346
  • 2
  • 11
3
votes
1 answer

Decompression does not work for own file

I'm realativly new to the python programming language and i ran into a problem with the module zstandard. I'm currently working with the replayfiles of Halite. Since they are compressed with zstandard, i have to use this module. And if i read a…
Noixes
  • 1,158
  • 12
  • 25
2
votes
1 answer

What is the most efficient way to unpack a .tar.zstd download in Python?

I am using the zstandard library to pack and unpack. I am essentially trying to recreate the following bash in Python: curl -O - http://download.com/file.tar.zst | zstd -d - | tar xf - I found this discussion saying that Python's tarfile module…
drowsily.throbbing872
  • 21
  • 2
2
votes
1 answer

ZStd. Heap corruption after decompression

In most cases it work good. For example, I compress 3D mesh. Almost all models compress\decompress good. But 2 or 3 models can give error when program try decompress it. It decompress good, but error when I free memory // how I read my files …
Тёма Басов
  • 33
  • 7
2
votes
1 answer

Decompressing tar.zst file-- why am I getting a premature end error?

I have been trying to download this dataset through my Mac terminal. I know it's huge! https://zenodo.org/record/3606810 I have the tar.zst file, and when I try to decompress (using zstd -d pol_0616-1119_labeled.tar.zst), it throws me this…
Ellen Jean Chapin
  • 21
  • 1
  • 1
  • 2
2
votes
0 answers

Error: "Decoding error (36) : Dictionary mismatch" using ZSTD decode on command line to decode a .ZST file

I am trying to decode a bunch of .ZST files which I do not know what the original file was so I can access them, however all of the .ZSTs return the exact same error and do not get decompressed. The error is Decoding error (36) : Dictionary…
jhm2k
  • 53
  • 9
2
votes
1 answer

Zstandard levels in hadoop

Compression level in org.apache.hadoop.io.compress.zstd.ZStandardCompressor does't seem to work. I see the reset function getting called in ZStandardCompressor constructor which is turn call init(level, stream) to call native function which I…
ondway
  • 114
  • 1
  • 11
1
vote
0 answers

"Decompressing large Zstandard file in Rust results in 'Frame requires too much memory for decoding' error"

I'm using the zstd-rs crate to decompress a large Zstandard file (~30GB) in Rust. My code opens the file, reads it into a buffer, and then uses the copy_decode function from the zstd::stream module to decompress the data and write it to an output…
weiserhase
  • 11
  • 3
1
2 3