how to work with 'while ..; do..; done' in bash

Question

I am having trouble using this command in bash to do the operation. The following is what I have:

cat namelist | while read line; do awk '$1 == $line {print $0}' Jan.txt; done

The namelist is a unrepeated namelist with each name on each line. And Jan.txt is a file of information like these:

ermartin pts/10       Tue Jan 19 09:49 - 10:15  (00:26)     171.66.208.139
sdou     pts/6        Tue Jan 19 01:52 - 22:20 (9+20:27)    131.243.186.99
sdou     pts/2        Tue Jan 19 01:20 - 22:20 (9+20:59)    131.243.186.99
phu      pts/2        Sat Jan 16 12:21 - 15:13  (02:52)     lbl-ipsec-7667.dhcp.lbl.gov
phu      pts/2        Sat Jan 16 10:07 - 12:01  (01:54)     lbl-ipsec-7622.dhcp.lbl.gov
sjames   pts/5        Fri Jan 15 12:06 - 13:06  (01:00)     lakshmi.lbl.gov
ermartin pts/2        Fri Jan 15 08:36 - 12:43  (04:07)     171.66.208.134
ermartin pts/2        Fri Jan 15 08:31 - 08:36  (00:05)     171.66.208.134
camitche pts/4        Thu Jan 14 15:18 - 20:47  (05:28)     171.66.208.134
ermartin pts/2        Thu Jan 14 13:20 - 15:57  (02:36)     171.66.208.139
ermartin pts/2        Thu Jan 14 12:35 - 13:00  (00:25)     171.66.208.139
sjames   pts/2        Thu Jan 14 06:17 - 06:27  (00:09)     phys1.als.lbl.gov
sjames   pts/2        Thu Jan 14 06:17 - 06:17  (00:00)     phys1.als.lbl.gov
ermartin pts/0        Wed Jan 13 15:19 - 15:50  (00:31)     171.66.208.139
ermartin pts/0        Wed Jan 13 07:55 - 08:09  (00:13)     c-24-130-14-154.hsd1.ca.comcast.net

I am just trying to match names in namelist to this file and then do the operation to each line of the information in this file but when I did $1 == $line, it says it's an error. How do I make use of this command properly?

Is it possible to also loop over the Months and print them as a title so like:

[Jan] 
sjames ...hours 
sdou ...hours 

[Feb] 
sjames ...hours 
sdou ...hours

Why would you run `awk` once per line, rather than having just one `awk` invocation process the whole of your input? — Charles Duffy, Jul 25 '16 at 18:56
btw, the problem is nothing to do with the `while read` loop; it's a problem with passing variables into `awk`. In `awk '$1 == $line'`, `awk` has no access to the contents to the shell variable `line` -- this would still be true even absent any `while read` loop. — Charles Duffy, Jul 25 '16 at 18:58
This also sounds like your try to "filter" the Jan.txt file for just the name in `namelist`. How about `grep -F -f namelist Jan.txt`. ? Good luck. — shellter, Jul 25 '16 at 18:58
Because for example after matching all the name with 'sdou' on the file, I wanna calculate the total duration the guy uses and then output to a new file with the name and the time used — Edccccch, Jul 25 '16 at 18:59
@Edccccch, ...I don't see how that answers the question. `awk` is a full-fledged scripting language -- the GNU version even supports date math, letting you properly calculate durations. There's no need to have bash involved in this script at all. — Charles Duffy, Jul 25 '16 at 19:00
The thing is that I need to do two for loop for the operation: — Edccccch, Jul 25 '16 at 19:20
like : for name in file: for the_name in namelist: if name == the_name: calculate the time for the name output to a file — Edccccch, Jul 25 '16 at 19:21
@Edccccch Awk is good at `for` loops and at calculating. I'm curious: in the elapsed time in Jan.txt, what does `9+` mean? — John1024, Jul 25 '16 at 19:26
It means 9 days and then hh:mm form. I figured out the pipe line that solves the time calculation. It's now just how do I make use of the for loop of awk that confuses me — Edccccch, Jul 25 '16 at 19:27
@Edccccch I just added to my answer code to calculate total times. — John1024, Jul 25 '16 at 19:45
BTW, if you really did want to implement this all in native bash, you could *also* quite easily read each line into a set of shell variables without needing `awk` at all (though the answer you have right now is great, so there's little need to replace it). — Charles Duffy, Jul 25 '16 at 19:46
I already showed you how to do this the right way (see http://stackoverflow.com/a/38543563/1745001), why are you now trying again to do it the wrong way? — Ed Morton, Jul 25 '16 at 20:20
@Ed I do not have GNU awk 4.* in my machine either on my work station which has linux system and this code is going to be run on the system that has the same set up as my machine so I figured it's better to work with the normal ways — Edccccch, Jul 25 '16 at 20:23
Then just make that comment in the original question so the answer you got that shows how to solve the problem can be tweaked to use POSIX-only syntax, don't open a whole new question and start all over again! — Ed Morton, Jul 25 '16 at 20:37

John1024 · Answer 1 · 2016-07-25T19:57:30.560

First, awk and shell do not share variables. It is, however, possible to transfer a shell variable to an awk variable using the -v option. The following works and the only change to your code is adding the -v line="$line" option:

cat namelist | while read line; do awk -v line="$line" '$1 == line {print $0}' Jan.txt; done

Note that awk references the line variable as just line with no dollar sign.

A possible replacement for the above pipeline is:

awk 'FNR==NR{a[$1]; next;} $1 in a' namelist Jan.txt

Here, the associative array a keeps track of what names were in namelist and uses it to decide whether to print lines in Jan.txt.

If you are were not going to do any more processing of the data (but I gather that the opposite is true), then, as Shellter suggests in the comments, the above could be replaced with:

grep -Ff namelist Jan.txt

If you want to calculate time used by users in namelist, then try:

$ awk -F'[ +:()]+' 'FNR==NR{a[$1]; next;} !($1 in a){next} NF==13{b[$1]+=$12+60*$11} NF==14{b[$1]+=$13+60*($12+24*$11)} END{for (n in b)print n,b[n]}' namelist Jan.txt

As an example, consider this namelist:

$ cat namelist 
sdou
phu

The calculated times are:

$ awk -F'[ +:()]+' 'FNR==NR{a[$1]; next;} !($1 in a){next} NF==13{b[$1]+=$12+60*$11} NF==14{b[$1]+=$13+60*($12+24*$11)} END{for (n in b)print n,b[n]}' namelist Jan.txt
phu 286
sdou 28406

For readability, it may be easier to see the same code spread over multiple lines:

awk -F'[ +:()]+' '
  FNR==NR{
    a[$1]
    next
  }

  !($1 in a){
    next
  }

  NF==13{
    b[$1]+=$12+60*$11
  }

  NF==14{
    b[$1]+=$13+60*($12+24*$11)
  }

  END{
    for (n in b)
      print n,b[n]
  }
  ' namelist Jan.txt
phu 286

To track usage by user and by month:

$ awk -F'[ +:()]+' 'FNR==NR{a[$1]; next;} !($1 in a){next} NF==13{b[$1"["$4"]"]+=$12+60*$11} NF==14{b[$1"["$4"]"]+=$13+60*($12+24*$11)} END{for (n in b)print n,b[n]}' namelist Jan.txt
phu[Jan] 286
sdou[Jan] 28406

I actually wanna do things like this: for name in file: for the_name in namelist: if name == the_name: calculate the time for the name and output to a file — Edccccch, Jul 25 '16 at 19:43
I figured out how to calculated time. It's just how do I have those for loops that matters and confuses me — Edccccch, Jul 25 '16 at 19:43
Is it possible to also loop over the Months and print them as a title so like: [Jan] sjames ...hours sdou ...hours [Feb] sjames ...hours sdou ...hours — Edccccch, Jul 25 '16 at 19:55

how to work with 'while ..; do..; done' in bash

1 Answers1