How to process fields between matching patterns?

Question

Related post: How to select lines between two patterns?

@fedorqui, thanks for providing all these different options for awk. I have been using this to parse through var log messages when troubleshooting ooms and it works great. I want to extend this further but I have not been able to figure out how to proceed. What I am trying to do:

Print the lines between rss and out of memory. I have done that with the example
Order the sections between each match by rss field. I have not been able to figure this out
Add an extra column with its own header and perform some mathematical operation. I have been able to do this somewhat but I am running into some formatting issues. I am not sure how to skip the first and last line when adding the column so I lose those lines. I am also not able to keep the spacing from the original if I do any operations other than print.

Here's the command I am using right now:

less /var/log/messages'|awk '/swapents/{x=1; print "=================="};/Out of memory/{x=0} x'|sed 's/[]\[]//g'

Here's the source Data:

Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.617265  pid    uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.622250  1828     0  1828     4331      116      14       3        0         -1000 udevd
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.627310  2664     0  2664    28002       53      23       3        0         -1000 auditd
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.633181  2680     0  2680    62032     1181      24       4        0             0 rsyslogd
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.638888  2694     0  2694     3444       61      11       3        0             0 irqbalance
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.644912  2710    81  2710     5430       56      14       3        0             0 dbus-daemon
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.651108  2779     0  2779    19958      203      42       3        0         -1000 sshd
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.656670  2789     0  2789     5622       56      17       3        0             0 xinetd
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.653452  Out of memory: Kill process 43390 (mysql) score 1000 or sacrifice child
blah
blah
blah
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.617265  pid    uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.622250  1828     0  1828     4331      116      14       3        0         -1000 udevd
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.627310  2664     0  2664    28002       53      23       3        0         -1000 auditd
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.633181  2680     0  2680    62032     1181      24       4        0             0 rsyslogd
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.638888  2694     0  2694     3444       61      11       3        0             0 irqbalance
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.644912  2710    81  2710     5430       56      14       3        0             0 dbus-daemon
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.651108  2779     0  2779    19958      203      42       3        0         -1000 sshd
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.656670  2789     0  2789     5622       56      17       3        0             0 xinetd
Sep  8 11:35:15 ip-10-23-15-70 kernel: 11810061.653452  Out of memory: Kill process 43390 (mysql) score 1000 or sacrifice child

Here's what my output looks like:

==================      0MB
Sep 8 11:35:15 pid  0MB name         <---- should be header (Pid virt rss etc)
Sep 8 11:35:15 1828 0MB udevd
Sep 8 11:35:15 2664 0MB auditd
Sep 8 11:35:15 2680 4MB rsyslogd
Sep 8 11:35:15 2694 0MB irqbalance
Sep 8 11:35:15 2710 0MB dbus-daemon
Sep 8 11:35:15 2779 0MB sshd
Sep 8 11:35:15 2789 0MB xinetd
Sep 8 11:35:15 2822 0MB crond
Sep 8 11:35:15 Out  0MB or           <---- should be footer (out of memory etc)
==================      0MB 
Sep 8 11:35:15 pid  0MB name         <---- should be header (Pid virt rss etc)
Sep 8 11:35:15 1828 0MB udevd
Sep 8 11:35:15 2664 0MB auditd
Sep 8 11:35:15 2680 4MB rsyslogd
Sep 8 11:35:15 2694 0MB irqbalance
Sep 8 11:35:15 2710 0MB dbus-daemon
Sep 8 11:35:15 2779 0MB sshd
Sep 8 11:35:15 2789 0MB xinetd
Sep 8 11:35:15 2822 0MB crond
Sep 8 11:35:15 Out  0MB or           <---- should be footer (out of memory etc)
==================      0MB

You can see the from the output that the separator I added for each oom field, awk tries to calculate values for it, I would love to avoid this if possible. Also the header and footer are getting chopped off and it would be nice to avoid that too.

Here's what I would like:

========================
Sep  8 11:35:15   pid   rss  memused_MB      oom_score_adj name
Sep  8 11:35:15   2664   53  {rss*4/1024}            -1000 auditd
Sep  8 11:35:15   2789   56  {rss*4/1024}                0 xinetd 
Sep  8 11:35:15   2710   56  {rss*4/1024}                0 dbus-dae
Sep  8 11:35:15   2694   61  {rss*4/1024}                0 irqbalan
Sep  8 11:35:15   1828  116  {rss*4/1024}            -1000 udevd 
Sep  8 11:35:15   2680  181  {rss*4/1024}                0 rsyslogd 
Sep  8 11:35:15   2779  203  {rss*4/1024}            -1000 sshd 
Sep  8 11:35:15   Out of memory: Kill process 43390 (mysql) score 1000 or sacrifice child
========================
Sep  8 11:35:15   pid   rss  memused_MB      oom_score_adj name
Sep  8 11:35:15   2664   53  {rss*4/1024}            -1000 auditd
Sep  8 11:35:15   2789   56  {rss*4/1024}                0 xinetd 
Sep  8 11:35:15   2710   56  {rss*4/1024}                0 dbus-dae
Sep  8 11:35:15   2694   61  {rss*4/1024}                0 irqbalan
Sep  8 11:35:15   1828  116  {rss*4/1024}            -1000 udevd 
Sep  8 11:35:15   2680  181  {rss*4/1024}                0 rsyslogd 
Sep  8 11:35:15   2779  203  {rss*4/1024}            -1000 sshd 
Sep  8 11:35:15   Out of memory: Kill process 43390 (mysql) score 1000 or sacrifice child
========================

Marc Lambrichs · Answer 1 · 2017-09-15T04:49:24.300

awk solution:

$ cat tst.awk
/swapents/ {
   x=1;
   print "=================="
   printf( "%s  %s %s   pid\t%4s\tmemused_MB\toom_score_adj\tname\n", $1, $2, $3, "rss");
   next
}
/Out of memory/ {
   printf( "%s  %s %s   %s\n", $1, $2, $3, substr($0,index($0,$7)));
   x=0
}
x {
   printf( "%s  %s %s   %s\t%4d\t%10.5f\t%13d\t%s\n", $1, $2, $3, $7, $11, ($11*4)/1024, $15, $16 )
}

You can play around with the formatting, like the precision in column 6, using the specifiers in the printf function. Call this with:

$ awk -f tst.awk /var/log/messages

EDIT: with sorting

OP asked to sort the output by rss columns. Using a standard sort wouldn't work here because you want to sort in between starting and ending matches. You can solve this by saving the intermediate result in an array and sort that with a self-defined function. Like this:

$ cat tst2.awk
/swapents/ {
    x=1;
    print "=================="
    printf( "%s  %s %s   pid\t%4s\tmemused_MB\toom_score_adj\tname\n", $1, $2, $3, "rss");
    next
}
/Out of memory/ {
    n=asort(a, sorted, "cmp_rss")
    for (i=1; i<=n; i++) {
        print sorted[i]
    }
    delete a;
    printf( "%s  %s %s   %s\n", $1, $2, $3, substr($0,index($0,$7)));
    x=0
}
x {
    a[i++] = sprintf( "%s  %s %s   %s\t%4d\t%10.5f\t%13d\t%s", $1, $2, $3, $7, $11, ($11*4)/1024, $15, $16 );
}
function cmp_rss(i1, v1, i2, v2)
{
    split(v1, a1, " ")
    split(v2, a2, " ")
    rss1=a1[5];
    rss2=a2[5];
    return (rss1 - rss2)
}

which leads to:

$ awk -f tst2.awk input.txt
==================
Sep  8 11:35:15   pid    rss    memused_MB  oom_score_adj   name
Sep  8 11:35:15   2664    53       0.20703          -1000   auditd
Sep  8 11:35:15   2710    56       0.21875              0   dbus-daemon
Sep  8 11:35:15   2789    56       0.21875              0   xinetd
Sep  8 11:35:15   2694    61       0.23828              0   irqbalance
Sep  8 11:35:15   1828   116       0.45312          -1000   udevd
Sep  8 11:35:15   2779   203       0.79297          -1000   sshd
Sep  8 11:35:15   2680  1181       4.61328              0   rsyslogd
Sep  8 11:35:15   Out of memory: Kill process 43390 (mysql) score 1000 or sacrifice child
==================
Sep  8 11:35:15   pid    rss    memused_MB  oom_score_adj   name
Sep  8 11:35:15   2664    53       0.20703          -1000   auditd
Sep  8 11:35:15   2710    56       0.21875              0   dbus-daemon
Sep  8 11:35:15   2789    56       0.21875              0   xinetd
Sep  8 11:35:15   2694    61       0.23828              0   irqbalance
Sep  8 11:35:15   1828   116       0.45312          -1000   udevd
Sep  8 11:35:15   2779   203       0.79297          -1000   sshd
Sep  8 11:35:15   2680  1181       4.61328              0   rsyslogd
Sep  8 11:35:15   Out of memory: Kill process 43390 (mysql) score 1000 or sacrifice child

hey awesome solution. This is what I used. /swapents/ { x=1; print "==================" gsub(/\[|\]/, "") print $0"\t\t"$11 next } {gsub(/\[|\]/, "")} /Out of memory/ { print $0 x=0 } x {printf "%s\t\t %10.2f\n", $0,$11*4/1024} This does not sort though. Is there a way to use this on command line without creating a file? I can't create files on the machine am using. — user8588010, Sep 15 '17 at 00:54
Added sorting example of rss column. Not creating a file? Sure, just `awk -f tst2.awk input.txt` doesn't create a file for you. — Marc Lambrichs, Sep 15 '17 at 04:40
Thanks for the sort method, thats pretty cool. By not creating a file, I meant I can't create the awk file on the machine I will be running this on. Looks like I may have to do the sorting externally :( . Thanks for the help, this will do for now, will live without sorting. — user8588010, Sep 15 '17 at 13:00
I can imagine you're not allowed access to a corporate production server. However, some companies use logservers where they allow read-access. And: there's no obligation, but you could accept the answer if you're really happy with it. — Marc Lambrichs, Sep 15 '17 at 15:05
yeah that is true but log servers only have logs from 2days before. So i would have to wait which doesn't work too well since the issue is happening now. — user8588010, Sep 15 '17 at 15:22

score 0 · Answer 2 · edited Sep 15 '17 at 05:58

Based on Marc Lambrichs response I was able to create this one liner that does the job. Thanks so much. The only thing missing now is sorting by the rss column, I haven't been able to get the rss field sorted though

less /var/log/messages|awk '/swapents/ {x=1; print "==================";gsub(/\[|\]/, "") ;printf "%s %s %s %s %10s %10s %10s %15s memory_used %-s\n", $1,$2,$3,$4,$7,$10,$11,$15,$16 ;next } {gsub(/\[|\]/, "")} /Out of memory/ {print $0 ;x=0 } x {printf "%s %s %s %s %10s %10s %10s %15s %9.2fMB %-s\n", $1,$2,$3,$4,$7,$10,$11,$15,$11*4/1024,$16}'

For readability, the awk code formatted:

/swapents/ {
    x=1; 
    print "==================";
    gsub(/\[|\]/, "");
    printf "%s %s %s %s %10s %10s %10s %15s memory_used %-s\n", $1,$2,$3,$4,$7,$10,$11,$15,$16 ;
    next 
} 
{
    gsub(/\[|\]/, "")
} 
/Out of memory/ {
    print $0;
    x=0 
} 
x {
    printf "%s %s %s %s %10s %10s %10s %15s %9.2fMB %-s\n", $1,$2,$3,$4,$7,$10,$11,$15,$11*4/1024,$16
}

Added sorting example to my answer. – Marc Lambrichs Sep 15 '17 at 04:41 — Marc Lambrichs, Sep 15 '17 at 04:41

How to process fields between matching patterns?

2 Answers2