Stack Overflow
Stack Overflow

Questions tagged [zstd]

Use with questions related to zstd library. This library is an open-source implementation of Zstandard, a lossless data compression algorithm developed by Yann Collet at Facebook.

91 questions
163
votes
6 answers

wbtc.cc.tar.zst

Suppose I have two files: file1.zst file2.tar.zst How can I decompress these files in terminal?
Quarkonia
  • 1,811
  • 2
  • 11
  • 11
10
votes
1 answer

Search through .zst text files

I have large zstd-compressed text files. How can I run fast search over them ? Can I use AG (The Silver Searcher) or something similar ? I've tried AG but it doesn't work, I have a "failed to load error" : ag -z -i "term"
titibouboul
  • 1,348
  • 3
  • 16
  • 30
9
votes
3 answers

How to change ZSTD compression level for files written via Spark?

It is stated that the default zstd compression level is 1 in Spark documentation. https://spark.apache.org/docs/latest/configuration.html I set this property to different values both in spark-defaults.conf, and inside the code like val conf = new…
belce
  • 203
  • 2
  • 5
7
votes
1 answer

zstd block compression in MongoDB/Docker

I use docker-compose under Windows 10 like so: version: '3' services: mongo: image: mongo:4.2 ports: - "27017:27017" restart: always volumes: - type: bind source: ${PWD}/mongod.conf target:…
Jonas Byström
  • 25,316
  • 23
  • 100
  • 147
7
votes
1 answer

Spark 3.0.1 tasks are failing when using zstd compression codec

I'm using Spark 3.0.1 with user provided Hadoop 3.2.0 and Scala 2.12.10 running on Kubernetes. Everything works fine when reading a parquet file compressed as snappy, however when I try to read a parquet file compressed as zstd several tasks fails…
phzz
  • 151
  • 1
  • 6
7
votes
6 answers

How to extract .zst files into a pandas dataframe

I'm a bit of a beginner when it comes to Python, but one of my projects from school needs me to perform classification algorithms on this reddit popularity dataset. The files are huge .zst files and can be found here:…
Aljen15
  • 77
  • 1
  • 1
  • 3
7
votes
1 answer

Could NOT find ZSTD (missing: ZSTD_LIBRARY ZSTD_INCLUDE_DIR)

I had an issue while installing headptrack with CMake when running cmake -DCMAKE_BUILD_TYPE=Release .. in heaptrack/build -- Could NOT find ZSTD (missing: ZSTD_LIBRARY ZSTD_INCLUDE_DIR) CMake Error at 3rdparty/libbacktrace/CMakeLists.txt:160…
Hugo Sohm
  • 2,872
  • 4
  • 23
  • 40
7
votes
1 answer

zstd: error 70 : Write error : Broken pipe (cannot write decoded block)

I'm trying through Windows terminal to decompress a large number of compressed files with zstd v1.4.0 and then 'ag' search over : zstd -dc -r . | ag -z -i "term" It gives me the following error while proceeding : zstd: error 70 : Write error :…
titibouboul
  • 1,348
  • 3
  • 16
  • 30
6
votes
1 answer

error when install mysqlclient using pip to MacOS Bigsur

I've tried common solutions written for mysqlclient error 1st try: brew install mysql-connector-c 2nd try: LDFLAGS=-L/usr/local/opt/openssl/lib pip install mysqlclient 3rd try: brew install zstd And mysql server is well-running. But the error is…
victory
  • 151
  • 2
  • 9
6
votes
3 answers

AZ64 compression format performance

AWS Redshift has recently released their own new encoding format AZ64, for which they say: Compared to ZSTD encoding, AZ64 consumed 5–10% less storage, and was 70% faster When I use an ANALYZE COMPRESSION my_table I still receive ZSTD as an…
Vzzarr
  • 4,600
  • 2
  • 43
  • 80
5
votes
2 answers

How to view zst file using Vim?

I'm trying to view some log files in zst format. I can use zstdcat to view the content, but when I do vim , there're only garbled text. Is there a similar way as zstdcat to view zst file with Vim as well?
Liutong Chen
  • 2,915
  • 5
  • 22
  • 29
4
votes
0 answers

Mercurial 5.9.3 on WinXP: Why zstd compression not available?

I'm in need for using Mercurial on WinXP 32bit. The repositories were created on a different system (Win10) and are relatively new, so they require revlog-compression-zstd. Unfortunately, this seems not to be available on the WinXP, even with…
Pastagod
  • 51
  • 2
4
votes
1 answer

How do I decompress large files using Zstd-jni and Byte Buffers

I am trying to decompress a lot of 40 MB+ files as I download them in parallel using ByteBuffers and Channels. I am getting better throughput by using Channels than I do by using Streams and we need this to be a very high throughput system as we…
Jay Askren
  • 10,282
  • 14
  • 53
  • 75
3
votes
0 answers

Stream a .zst compressed file line by line

I am trying to sift through a big database that is compressed in a .zst. I am aware that I can simply just decompress it and then work on the resulting file, but that uses up a lot of space on my ssd and takes 2+ hours so I would like to avoid that…
SimonUnderwood
  • 469
  • 3
  • 12
3
votes
1 answer

How to change zstd level for qcow2 image?

There is an option compression_type=zstd to enable zstd compression for qcow2 according to wiki. But it always uses zstd level 3, how could I do to compress it with level 19? This image is read-only and I just want the max compression level. The…
kkocdko
  • 300
  • 2
  • 8
1
2 3 4 5 6 7