Bash - Find variable in many .txt files and calculate statistics

Question

I have many .txt files in a folder. They are full of statistics, and have a name that's representative of the experiment those statistics are about.

exp_1_try_1.txt
exp_1_try_2.txt
exp_1_try_3.txt

exp_2_try_1.txt
exp_2_try_2.txt

exp_other.txt

In those files, I need to find the value of a variable with a specific name, and use them to calculate some statistics: min, max, avg, std dev and median.

The variable is a decimal value and dot "." is used as a decimal separator. No scientific notation, although it would be nice to handle that as well.

#in file exp_1_try_1.txt
var1=30.523
var2=0.6

#in file exp_1_try_2.txt
var1=78.98
var2=0.4

#in file exp_1_try_3.txt
var1=78.100
var2=1.1

In order to do this, I'm using bash. Here's an old script I made before my bash skills got rusty. It calculates the average of an integer value.

#!/bin/bash

folder=$1
varName="nHops"

cd "$folder"
grep -r -n -i --include="*_out.txt" "$varName" . | sed -E 's/(.+'"$varName"'=([0-9]+))|.*/\2/' | awk '{count1+=$1; count2+=$1+1}END{print "avg hops:",count1/NR; print "avg path length:",count2/NR}' RS="\n"

I'd like to modify this script to:

• support finding decimal values of variable length
• calculate more statistics

In particular std dev and median may require special attention.

Update: Here's my try to solve the problem using only UNIX tools, partially inspired by this answer. It works fine, except it does not calculate the standard deviation. The chosen answer uses Perl and is probably much faster.

#!/bin/bash

folder=$1
varName="var1"

cd "$folder"
grep -r -n -i --include="exp_1_run_*" "$varName" . | sed -E 's/(.+'"$varName"'=([0-9]+(\.[0-9]*)?))/\2/' | sort -n | awk '
  BEGIN {
    count = 0;
    sum = 0;
  }
  {
    a[count++] = $1;
    sum += $1;
  }
  END {
    avg = sum / count;
    if( (count % 2) == 1 ) {
      median = a[ int(count/2) ];
    } else {
      median = ( a[count/2] + a[count/2-1] ) / 2;
    }
    OFS="\t";
    OFMT="%.6f";
    print avg, median, a[0], a[count-1];
  }
'

Answer 1

To extract just the values, use the -o and -P grep options:

grep -rioPh --include="*_out.txt" "(?<=${varName}=)[\d.]+" .

That looks for a pattern like nHops=1.234 and just prints out 1.234

Given your sample data:

$ var="var1"
$ grep -oPh "(?<=$var=)[\d.]+" exp_1_try_{1,2,3}.txt 
30.523
78.98
78.100

To output some stats, you should be able to pipe those numbers into your favourite stats program. Here's an example:

grep -oPh "(?<=$var=)[\d.]+" f? | 
perl -MStatistics::Basic=:all -le '
    @data = <>; 
    print "mean: ", mean(@data);
    print "median: ", median(@data);
    print "stddev: ", stddev(@data)
'

mean: 62.53
median: 78.1
stddev: 22.64

---

Of course, since this is perl, we don't need grep or sed at all:

perl -MStatistics::Basic=:all -MList::Util=min,max -lne '
        /'"$var"'\s*=\s*(\d+\.?\d*)/ and push @data, $1
    } END {
        print "mean: ", mean(@data);
        print "median: ", median(@data);
        print "stddev: ", stddev(@data);
        print "min: ", min(@data);
        print "max: ", max(@data);
' exp_1_try_*

mean: 62.53
median: 78.1
stddev: 22.64
min: 30.523
max: 78.98