1

Following this 2010 question Gzip with all cores, I would like to gzip files using multiple core and indicate a progress bar with pv tool.

How do I improve this code?

CORES=$(grep -c '^processor' /proc/cpuinfo)
find /source -type f -print0 | xargs -0 -n 1 -P $CORES gzip -9

I would like to show remaining time and show the progression bars running in parallel.

Do you have any other best alternatives as of 2018?

Thanks.

Cyrus
  • 84,225
  • 14
  • 89
  • 153
Michel Hua
  • 1,614
  • 2
  • 23
  • 44
  • 1
    Check GNU parallel -> https://unix.stackexchange.com/questions/182623/how-to-use-gnu-parallel-effectively – Samuel Aug 26 '18 at 09:58
  • With parameters `-c` and `-N` you can have more than one pipeview running in parallel. In any case, you are not running `gzip` with all cores. You are running many `gzip`, one in each core. To run the gzip algorithm in many cores, use `pigz`. – Poshi Aug 26 '18 at 10:26

2 Answers2

4

Use GNU Parallel which has a progress bar or an eta:

find ... -print0 | parallel -0 --progress gzip -9 {}

Or

find ... -print0 | parallel -0 --eta ...

Or

find ... -print0 | parallel -0 --bar ...
Mark Setchell
  • 191,897
  • 31
  • 273
  • 432
  • This is my preferred solution, too. Tip if you're using all your cores with something CPU intensive like `gzip -9`, you might want to `nice` things a bit. You could slip a `nice -n10` either immediately before the `gzip`, or else immediately before the `parallel`, and all the child pids inherit the niceness of parallel. Alternatively, you can run parallel with one less threads than you have cores with `parallel -j$(($(nproc)-1))`, sacrificing a little throughput. I just prefer the `nice`ing method, so you're fully employing idle resources but yielding when normal tasks want CPU time. – Joshua Huber Sep 26 '18 at 20:40
0

You're not really running gzip on multiple cores, you're running multiple gzip's.

pv runs on a pipe. I don't believe it's possible to have it run on multiple pipes at the same time and provide a summary of all of them.

Evert
  • 93,428
  • 18
  • 118
  • 189
  • I am tracking progress with `htop` with the CPU bars, but it doesn't provide the visually feedback `pv` offers – Michel Hua Aug 27 '18 at 17:57