53

So I am using pigz in tar with

tar --use-compress-program=pigz

and this works but it doesn't use all of my processors, and I'd like to make sure it's recursive (-r) and using (-9 compression level).

I read through Utilizing multi core for tar+gzip/bzip compression/decompression but it doesn't note anywhere to add additional commands in that format, and I couldn't find anything in the man page for either program for additional swithed.

Thanks, Cam

Community
  • 1
  • 1
Cam
  • 545
  • 1
  • 4
  • 4
  • Possible duplicate of [Utilizing multi core for tar+gzip/bzip compression/decompression](http://stackoverflow.com/questions/12313242/utilizing-multi-core-for-targzip-bzip-compression-decompression) – luchaninov Jan 15 '17 at 09:53
  • AFAIK gzip level 9 is not a good choice, as it takes much more CPU times without much benefit. YMMV, but as compared to level 4 you have 3 times more CPU time for about 1% file size difference. Also, xz level 1 has less calculations than gz level 9 with better rates. – bct Jun 13 '19 at 01:48

3 Answers3

63

Mark Adler's top voted answer on the SO link that you included in your question does provide a solution for specifying compression-level as well as number of processors to use:

tar cf - paths-to-archive | pigz -9 -p 32 > archive.tar.gz

See : https://stackoverflow.com/a/12320421

Community
  • 1
  • 1
Amit.Sinha
  • 746
  • 7
  • 6
49

To pass arguments to pigz using -I or --use-compress-program, you can enclose the command and arguments in quotes, like so:

tar --use-compress-program="pigz --best --recursive" -cf archive.tar.gz YourData

Here's a fun option to monitor the speed of the archive creation:

tar --use-compress-program="pigz --best --recursive | pv" -cf archive.tar.gz YourData
Stefan Lasiewski
  • 17,380
  • 5
  • 28
  • 35
  • 2
    Can this command be used like this, `tar --use-compress-program="pigz -dc -p 20" -xvpf test.tgz -C /home/test_dir`? I tried, but it **failed**. I want to decompress my tgz file under number of custom threads. – CHENJIAN Nov 02 '18 at 12:05
  • pigz cannot parallelise decompression, according to the man page [1], but does split out reading, writing, and checking into separate threads so is at least a bit faster than gzip. [1] https://linux.die.net/man/1/pigz – Chris L. Barnes Dec 09 '20 at 11:50
34

fast unpack:

tar -I pigz -xf /mnt/sd/current/backup/bigbackup_web.tar.gz -C /tmp

fast pack:

tar -cf bigbackup.tar.gz -I pigz /opt

then:

apt-get install pigz

or

yum install pigz

or for MacOS with Homebrew

brew install pigz
davibusanello
  • 3
  • 2
Nick Kutuzov
  • 457
  • 4
  • 3
  • 2
    The argument `--use-compress-program=pigz` in the OP is the same as `-I pigz`. – akhan Dec 12 '19 at 22:16
  • Can we speed up the unpacking process? As far as i know, compressing doesn't need any extra arguments and pigz uses every core. But when i unpack it, it doesn't seem to be using all cpu... – postgresnewbie Apr 20 '22 at 05:43