Bash Text Parsing

Question

I am getting confused with parameter substitution. Basically the file to be parsed is in the following structure:

foo.txt:
    system.switch_cpus.commit.op_class_0::total     10000000                       # Class of committed instruction
    system.switch_cpus.commit.bw_lim_events      10000000                       # number cycles where commit BW limit reached
    system.switch_cpus.rob.rob_reads             80558432                       # The number of ROB reads
    system.switch_cpus.rob.rob_writes            43430539                       # The number of ROB writes
    system.switch_cpus.timesIdled                   37218                       # Number of times that the entire CPU went into an idle state and unscheduled itself
    system.switch_cpus.idleCycles                 2755508                       # Total number of cycles that the CPU has spent unscheduled due to idling
    system.switch_cpus.committedInsts            10000000                       # Number of Instructions Simulated
    system.switch_cpus.committedOps              10000000                       # Number of Ops (including micro ops) Simulated
    system.switch_cpus.cpi                       8.369191                       # CPI: Cycles Per Instruction
    system.switch_cpus.cpi_total                 8.369191                       # CPI: Total CPI of All Threads
    system.switch_cpus.ipc                       0.119486                       # IPC: Instructions Per Cycle
    system.switch_cpus.ipc_total                 0.119486                       # IPC: Total IPC of All Threads
    system.switch_cpus.int_regfile_reads         21773538                       # number of integer regfile reads
    system.switch_cpus.int_regfile_writes         9447282                       # number of integer regfile writes

I want to find the following variables and print out the corresponding value:

    list=(IPC CPI)
IPC="system.switch_cpus.ipc"
CPI="system.switch_cpus.cpi"

for i in $list:
do 
awk -v a="$i" '{$1 == $a} {print}' $1
done

Then I run the script with the following command:

./parser.sh foo.txt

This is printing out the whole file.

Output:
system.switch_cpus.commit.op_class_0::total     10000000                       # Class of committed instruction
system.switch_cpus.commit.bw_lim_events      10000000                       # number cycles where commit BW limit reached
system.switch_cpus.rob.rob_reads             80558432                       # The number of ROB reads
system.switch_cpus.rob.rob_writes            43430539                       # The number of ROB writes
system.switch_cpus.timesIdled                   37218                       # Number of times that the entire CPU went into an idle state and unscheduled itself
system.switch_cpus.idleCycles                 2755508                       # Total number of cycles that the CPU has spent unscheduled due to idling
system.switch_cpus.committedInsts            10000000                       # Number of Instructions Simulated
system.switch_cpus.committedOps              10000000                       # Number of Ops (including micro ops) Simulated
system.switch_cpus.cpi                       8.369191                       # CPI: Cycles Per Instruction
system.switch_cpus.cpi_total                 8.369191                       # CPI: Total CPI of All Threads
system.switch_cpus.ipc                       0.119486                       # IPC: Instructions Per Cycle
system.switch_cpus.ipc_total                 0.119486                       # IPC: Total IPC of All Threads
system.switch_cpus.int_regfile_reads         21773538                       # number of integer regfile reads
system.switch_cpus.int_regfile_writes         9447282                       # number of integer regfile writes

How can create a list of variables in shell who have their own values and parse each of them from a file using awk or sed?

maybe a simple egrep would do it? `egrep -iw 'system.switch_cpus.ipc|system.switch_cpus.cpi' foo.txt` — Vinicius Placco, Feb 27 '18 at 17:18
Expected output is the whole line or the second variable that contains the value for cpi in "foo.txt". I can change that setting in teh awk later on i think. — user3285014, Feb 27 '18 at 17:20
@ViniciusPlacco: Unfortunately I have a growing list of variables. In this particular question, I have shown only 2 variables. The egrep option is cool, but when to search for 10s of variables, it might get tricky and error prone for me :-( — user3285014, Feb 27 '18 at 17:22
OK, I see! You may get some info on `grep`ing a list of words from this post: https://stackoverflow.com/questions/17863301/how-to-grep-with-a-list-of-words. Good luck! — Vinicius Placco, Feb 27 '18 at 17:25
@Luis Munoz you have to erase the `if`. `awk -v a="$i" '($1 == $a){print $0}' $1` — Arno Bozo, Feb 27 '18 at 17:31

karakfa · Accepted Answer · 2018-02-27T18:14:28.953

you can do all in one awk script, if your list is not in a file you can use a here document and file substitution as below.

$ awk 'NR==FNR{a[$1]; next} $1 in a' <(cat << EOF
system.switch_cpus.ipc
system.switch_cpus.cpi
EOF
) file

will give you

system.switch_cpus.cpi                       8.369191                       # CPI: Cycles Per Instruction
system.switch_cpus.ipc                       0.119486                       # IPC: Instructions Per Cycle

If you want to search one at a time with a variable

$ var='system.switch_cpus.ipc'; awk -v var="$var" '$1==var' file

system.switch_cpus.ipc                       0.119486                       # IPC: Instructions Per Cycle

However, in that case, using grep might be better off

$ var='system.switch_cpus.ipc'; grep -wF "$var" file

system.switch_cpus.ipc                       0.119486                       # IPC: Instructions Per Cycle

UPDATE

If your variable names are in a list, you can decode the values with this

$ vars=(var1 var2)        # define the list with variables, values even may not be assigned yet
$ var1=value1; var2=value2                   # assign values
$ for v in ${vars[@]}; do echo ${!v}; done   # extract values with indirect reference
value1
value2

I need to run with for loop because I have around 15 variables to look for in a file. The file option might work. But I there is no option in shell to substitute the parameter (Like the list element "IPC" has a value "system.switch_cpus.ipc" which the awk should find)? Basically I was trying to see if there is two time paramater substitution option in shell. But, your option would also work for me. Thanks. — user3285014, Feb 27 '18 at 18:01
Thanks @karakfa. I was looking for the variable indirection. I just did and now its working as I wanted. — user3285014, Feb 27 '18 at 18:42

score 0 · Answer 2 · answered Feb 27 '18 at 17:51

0

This code will print the entire record

awk -v a="$i" '{ if($1 == a){print $0} }' $1

One of the problems with your code is that variable a must not be preceded by $ in awk.

answered Feb 27 '18 at 17:51

LMC

10,453
2
27
52

`awk -v a="$i" '$1==a' $1` will do the same and more idiomatic. – karakfa Feb 27 '18 at 18:06
Well, it's less readable. Not everyone knows as much awk as you do ;-) – LMC Feb 27 '18 at 20:12

Bash Text Parsing

2 Answers2