1

I have 10 files with the name of

data_00
data_01
data_02
...
data_09

The first 8 lines of the data files look like the following:

Stamp_number
10
Item_number
9000
Position
5.1008068168967009e+00 5.4899193183110690e+01
5.1008068168967009e+00 5.4899193183110690e+01
5.1008068168967009e+00 5.4899193183110690e+01

All 10 files have the same format but different values for all numbers.

I wish to assign a bash variable using awk for the values of the 6th, 7th, and 8th lines of the 1st and 2nd columns of 10 files and get the difference between them.

for i in {00..09}; do
a=$(awk 'NR==6 {print $1}' data_$i)
b=$(awk 'NR==6 {print $2}' data_$i)
c=$(awk 'NR==7 {print $1}' data_$i)
d=$(awk 'NR==7 {print $2}' data_$i)
e=$(awk 'NR==8 {print $1}' data_$i)
f=$(awk 'NR==8 {print $2}' data_$i) 
val_ab=`bc -l <<< "$a-$b"`
val_cd=`bc -l <<< "$c-$d"`
val_ef=`bc -l <<< "$e-$f"`
echo $val_ab
echo $val_cd
echo $val_ef
done

But this prints out the following syntax error 10 times:

(standard_in) 1: syntax error
(standard_in) 1: syntax error
(standard_in) 1: syntax error
(standard_in) 1: syntax error
(standard_in) 1: syntax error
(standard_in) 1: syntax error
0.000000
0.000000
0.000000

I tried data_"$i", data_[$i], data_${i} but all fails. How can I use the for-loop index i (or any other bash variable) inside the awk command substitution to assign a bash variable?

exsonic01
  • 615
  • 7
  • 27
  • Which of the commands in your loop does actually produce the syntax error, and for which value for `i`? You get 6 error messages, the loop is executed 100 times, and you have inside the loop only 3 commands which read something from standard input. This somehow does not fit together? – user1934428 Jun 02 '21 at 05:39
  • I also suggest that you add a `echo using $BASH_VERSION` before entering the loop and let us now what you get as a result. – user1934428 Jun 02 '21 at 05:40
  • @user1934428 It says using 4.2.46(2)-release I don't know exactly where the error comes from, but my guess is the use of i in the data_$i in the command substitution. – exsonic01 Jun 02 '21 at 06:21
  • 1
    This is easy to find out! Just run your program with `set -x` enabled and have a look! It does not make sense to discuss an error, if we don't even know where it comes from. – user1934428 Jun 02 '21 at 06:24
  • @user1934428 Thanks to let me know. Well, I was looking at the wrong part. It was the bc lines. – exsonic01 Jun 02 '21 at 06:28
  • And what is actually passed to `bc`? You really should post the relevant information on your own, and not wait until someone asks you. This is a simple debugging job, you already have the output of `set -x`, so why don't you show what you have? BTW, it may also make sense to post your version of bc, and whether you run on Linux or BSD or whatever. – user1934428 Jun 02 '21 at 06:33
  • @user1934428 val_ab=`bc -l <<< "$a-$b"` these lines were the issue. I tried to calculate the difference between a and b and assign them. That was all. I used similar command a long ago, but for some reason this time it does not work. And I really didn't know about debugging option -x... Thanks to let me know about this. – exsonic01 Jun 02 '21 at 06:43
  • This does not show what **really** has been fed into `bc`, because we don't see the result of the parameter expansion! – user1934428 Jun 02 '21 at 06:46
  • @user1934428 Sorry but I fail to understand your comment... I tried to calculate the simple math, $a minus $b. All 6 values and the difference of them are used in other later parts of my bash script but these later parts are OK. It is the "bc" lines that were creating error. – exsonic01 Jun 02 '21 at 06:56
  • I need to see the **expanded** bc lines, not your source code. Ah, this doesn't lead anywhere. OK, you run your program with `set -x` enabled. This shows a trace of the program. Now you grab your mouse and copy the output of the trace of the erroneous call, and paste it in your question. This is the information one needs to investigate further. – user1934428 Jun 02 '21 at 07:01

2 Answers2

5

bc does not support scientific notation but awk does

awk 'NR>=6 && NR<=8 {print $1-$2}' "data_$i"
Diego Torres Milano
  • 65,697
  • 9
  • 111
  • 134
  • Also, the clumsy and verbose repeated Awk invocations in your loop really really beg you to refactor them into a single Awk script like this. If the files are large, this could be optimized by adding an `exit` when you reach line number 8. – tripleee Jun 02 '21 at 04:31
  • It's not at all clear from your example that this is a requirement. Probably extend the Awk script to do the remainder of the processing from inside the loop; or have Awk print all nine at once into a Bash array: `result=($(awk 'NR >=6 && NR <= 8 { print $1, $2, $1-2 } NR == 8 { exit }' "data_$i"))` – tripleee Jun 02 '21 at 06:36
  • @tripleee Thanks to let me know. In your example, can I approach every 9 elements of the array "result" using result[1] ~ result[9]? – exsonic01 Jun 02 '21 at 06:58
  • Bash arrays are zero-based and the syntax looks slightly different. It's pretty ugly for backwards-compatibility reasons (remember Bash tries to be Bourne-compatible, so anything outside Bourne needs to use syntax which doesn't already mean something in Bourne shell.) The first element is `${result[0]}` and the final one is `${result[8]}` – tripleee Jun 02 '21 at 07:07
  • @tripleee Thank you. Can I use ${result[0]} ~ ${result[8]} in awk or sed command substitution inside the bash script, just like other bash variables inside the awk or sed in the bash script? – exsonic01 Jun 02 '21 at 20:35
  • See https://stackoverflow.com/questions/19075671/how-do-i-use-shell-variables-in-an-awk-script – Diego Torres Milano Jun 02 '21 at 20:59
0

You could convert the E-notation to 10^ before the calculation, e.g.:

read a b c < <(tail -n3 data_00            |
               sed 's/e+\?/*10^/g; s/ /-/' |
               bc -l                       |
               tr '\n' ' ')
echo $a $b $c

Output:

-49.7983863662139891
-49.7983863662139891
-49.7983863662139891
Thor
  • 45,082
  • 11
  • 119
  • 130