1

I found this great progress bar: Great Example

How can I implement this progress bar when curl running and do sort and unique etc on really big file (60GB) taking about ~ 3 hours.

Example: count="$(awk '{print $1}' $FILE_NAME | sort -T /diskXX --parallel=$PARALLEL | uniq | wc -l)"

Thanks! Guy

Cyrus
  • 84,225
  • 14
  • 89
  • 153
G.Guy
  • 19
  • 3

2 Answers2

1

With pv:

pv "$FILE_NAME" | awk '{print $1}' | sort -T /diskXX --parallel="$PARALLEL" | uniq | wc -l

or from stdin with size in gigabyte:

pv -s 60g <"$FILE_NAME" | awk '{print $1}' | sort -T /diskXX --parallel="$PARALLEL" | uniq | wc -l

From man pv:

pv shows the progress of data through a pipeline by giving information such as time elapsed, percentage completed (with progress bar), current throughput rate, total data transferred, and ETA

Cyrus
  • 84,225
  • 14
  • 89
  • 153
  • How can I run `pv` and save to variable: `count=pv "$FILE_NAME" | awk '{print $1}' | sort -T /diskXX --parallel="$PARALLEL" | uniq | wc -l`? Thank YOU! Great :) – G.Guy May 01 '18 at 10:35
  • Usually like this: `count=$(pv "$FILE_NAME" | awk '{print $1}' | sort -T /diskXX --parallel="$PARALLEL" | uniq | wc -l)` – Cyrus May 01 '18 at 10:45
  • Unfortunately `pv` should be installed separately. So I go back to thinking about how to implement the first option in code – G.Guy May 01 '18 at 11:08
  • Here is a [pure bash progress bar](https://stackoverflow.com/a/68298090/1765658) – F. Hauri - Give Up GitHub Aug 30 '22 at 07:19
0

One option is to implement your own progress visualization function, by printing a status (or progress bar) in the same line (i.e. overwriting the previous content of the line).

For example you can use "echo -n" in a loop to always print the assumed runtime percentage of a background process (10%, 20%, 30%).

Edit:
You might find this question and collection of answers helpful.

pitseeker
  • 2,535
  • 1
  • 27
  • 33
  • Thank you for your feedback! I don't use a look, I need to run a script in one line that will be a process. I do not know how to add this to a progress bar :) – G.Guy May 03 '18 at 06:11