50

Is there a way to spread xz compression efforts across multiple CPU's? I realize that this doesn't appear possible with xz itself, but are there other utilities that implement the same compression algorithm that would allow more efficient processor utilization? I will be running this in scripts and utility apps on systems with 16+ processors and it would be useful to at least use 4-8 processors to potentially speed up compression rates.

ylluminate
  • 12,102
  • 17
  • 78
  • 152

1 Answers1

73

Multiprocessor (multithreading) compression support was added to xz in version 5.2, in December 2014.

To enable the functionality, add the -T option, along with either the number of worker threads to spawn, or -T0 to spawn as many CPU's as the OS reports:

xz -T0 big.tar
xz -T4 bigish.tar

The default single threaded operation is equivalent to -T1.

I have found that running it with a couple of hyper-threads less than the total number of hyperthreads on my CPU† provides a good balance of responsiveness and compression speed.

† So -T10 on my 6 core, 12 thread workstation.

As scai and Dzenly said in comments

If you want to use this in combination with tar just call export XZ_DEFAULTS="-T 0" before.

or use smth like: XZ_OPT="-2 -T0"

Mark Booth
  • 7,605
  • 2
  • 68
  • 92
  • 18
    If you want to use this in combination with `tar` just call `export XZ_DEFAULTS="-T 0"` before. – scai Dec 28 '18 at 12:30
  • 1
    or use smth like: XZ_OPT="-2 -T0" – Dzenly Jun 08 '19 at 14:16
  • 1
    I used the option XZ -T4 but I just see only one processor being used. – Luciano Andress Martini Dec 05 '19 at 14:14
  • Sorry @LucianoAndressMartini I don't know what to suggest. If you're using version 5.2 or later (check with `xz -V`) and `-T4` doesn't work on large files which aren't trivial to compress, I would suggest asking a question on [superuser.se] detailing your computer, operating system and kind of data you're compressing, – Mark Booth Dec 06 '19 at 11:36
  • 4
    It seems that decompressing with multiple cores doesn't work, i.e. `xz --decompress -T0 file.tar.xz`. Is the decompression fundamentally non-parallelizable with xz? – jmon12 Jul 01 '20 at 08:18
  • 6
    Threading is all about speeding up compression @jmon12 , as decompression is trivial by comparison. From the man page "Threaded decompression hasn't been implemented yet. It will only work on files that contain multiple blocks with size information in block headers. All files compressed in multi-threaded mode meet this condition, but files compressed in single-threaded mode don't, even if `--block-size=size` is used." – Mark Booth Jul 10 '20 at 16:31
  • So supposing I want tar to do multi thread xz compression I could do something like this: tar cpO somename | xz -zT 5 -c > somename.tzx or alternatively set the env variable XZ_DEFAULTS="-T 5" (or whatever number you think is most appropriate for your system .... 0 will probably allow you to do little else until you finish the compression) – louigi600 Sep 07 '21 at 06:07
  • @jmon12 pixz supports multi-threaded decompression – Étienne Jan 25 '23 at 13:56