1

I'm using GNU Parallel to speed up downloading of several thousand files during a build process running in docker container. This operation is lengthy and I want to have a progress bar that will periodically log that everything is going OK. This process will also be run asynchronously and logs will be gathered in CloudWatch. Preferably I'd like to use one of existing tools instead of coming up with custom solution that will print $(find output_dir -type f | wc -l ) / $numfiles .

I've used --bar parameter for parallel but instead of printing a line on every change (as it does when run outside of docker) but only the last "line" (starting with 100%) was printed, possibly because only the last line contained a new line. I've also tried to use pv (especially with --interval flag that would allow me to log progress only once a minute or so) but the result was the same.

Is there any other tool I can use to achieve my goal? Or maybe I can somehow make parallel work?

tripleee
  • 175,061
  • 34
  • 275
  • 318
MatXXX
  • 31
  • 2
  • Docker does weird things to the terminal. How exactly are you running it? – tripleee Mar 30 '23 at 16:44
  • @tripleee locally just using `docker run`, without `-it` etc. In the end I'll run this script on ECS thus I want it to work with default terminal options. – MatXXX Mar 30 '23 at 16:49

1 Answers1

-1

I get a nice bar by running:

host$ docker run -it ubuntu:xenial /bin/bash
docker$ apt update
docker$ apt install parallel
docker$ parallel --bar true ::: {1..1000}

I get the same by running (except it seems the terminal width is detected as 80 chars):

docker run ubuntu:xenial /bin/bash -c 'apt update;apt install -y parallel; parallel --bar true ::: {1..1000}'

So it seems parallel has no problem giving you a bar - even when running inside Docker.

Ole Tange
  • 31,768
  • 5
  • 86
  • 104