Replacing multi line with another multi line using AWK

Question

I have Some markdown text that I want to have replaced everytime I run a Shell script to get kubernetes information. Here is that markdown:

# Kube_cluster
| privateVlan   | publicVlan   | privateIP   | publicIP   | machineType   | location   | id   | state   | status   | statusDate   | statusDetails   | errorMessage   | errorMessageDate   | isolation   | kubeVersion   | targetVersion   | reasonForDelete   | versionEOS   | masterVersionEOS   | trustedStatus   | poolid   | poolName   | pendingOperation    |
| --------------: | :------------- | :------------ | :----------- | :-------------- | :----------- | :----- | :-------- | :--------- | :------------- | :---------------- | :--------------- | :------------------- | :------------ | :-------------- | :---------------- | :------------------ | :------------- | :------------------- | :---------------- | :--------- | :----------- | :-------------------
9999999 | 6966666 | 0.0.0.212 | 10.257.257.42 | b3c.16x64.encrypted | dal10 | Kube-0000067d | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | private | 1.23.14_1553 | 1.23.14_1554 |  |  |  | unsupported | okiedokie-1e501db | default | 
9999999 | 6966666 | 0.0.0.195 | 10.257.257.82 | b3c.16x64.encrypted | dal10 | Kube-00001ab8 | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | private | 1.23.14_1553 | 1.23.14_1554 |  |  |  | unsupported | okiedokie-1e501db | default | 
9999999 | 6966666 | 0.0.0.249 | 10.257.257.74 | b3c.16x64.encrypted | dal10 | Kube-00001c81 | normal | Ready | 2022-12-08T22:22:32+0000 |  |  |  | private | 1.23.14_1553 | 1.23.14_1554 |  |  |  | unsupported | okiedokie-1e501db | default | 
9999999 | 6966666 | 0.0.0.242 | 10.257.257.85 | b3c.16x64.encrypted | dal10 | Kube-00001d6b | normal | Ready | 2022-12-08T22:22:32+0000 |  |  |  | private | 1.23.14_1553 | 1.23.14_1554 |  |  |  | unsupported | okiedokie-1e501db | default | 
9999999 | 6966666 | 0.0.0.194 | 10.257.257.90 | b3c.16x64.encrypted | dal10 | Kube-00001ea3 | normal | Ready | 2022-12-08T22:22:29+0000 |  |  |  | private | 1.23.14_1553 | 1.23.14_1554 |  |  |  | unsupported | okiedokie-1e501db | default | 
9999999 | 6966666 | 0.0.0.251 | 10.257.257.66 | b3c.8x32.encrypted | dal10 | Edge-00001fde | normal | Ready | 2022-12-08T22:22:32+0000 |  |  |  | public | 1.23.14_1553 | 1.23.14_1554 |  |  |  | unsupported | okiedokie-de32bdc | edge | 
9999999 | 6966666 | 0.0.0.198 | 10.257.257.68 | b3c.8x32.encrypted | dal10 | Edge-00002009 | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | public | 1.23.14_1553 | 1.23.14_1554 |  |  |  | unsupported | okiedokie-de32bdc | edge | 
9999999 | 6966666 | 0.0.0.207 | 10.257.257.94 | b3c.8x32.encrypted | dal10 | Edge-000021fa | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | public | 1.23.14_1553 | 1.23.14_1554 |  |  |  | unsupported | okiedokie-de32bdc | edge | 



## Addon Status
OK
Name                  Version                Health State   Health Status   
addon1   1.0.0                  normal         Addon Ready.  
addon2                 1.14* (1.16 default)   normal        


---<br><br>


test

The object of this script is to replace the markdown table with another markdown table that has newer information.

I can find the Text i want to search for using this awk command (From the name of the cluster to --- ):

test=Kube_cluster
replace=$(awk  'BEGIN {p=0};
/'$test'/ {p=1 ;next};
/---<br><br>/ {p=0;next};
p {print }' "md/$test.md" )

Now I run script again and I get different Markdown. That I want to replace in line the previous Markdown ( I want to replace the whole markdown block each run ):

# Kube_cluster
| privateVlan   | publicVlan   | privateIP   | publicIP   | machineType   | location   | id   | state   | status   | statusDate   | statusDetails   | errorMessage   | errorMessageDate   | isolation   | kubeVersion   | targetVersion   | reasonForDelete   | versionEOS   | masterVersionEOS   | trustedStatus   | poolid   | poolName   | pendingOperation    |
| --------------: | :------------- | :------------ | :----------- | :-------------- | :----------- | :----- | :-------- | :--------- | :------------- | :---------------- | :--------------- | :------------------- | :------------ | :-------------- | :---------------- | :------------------ | :------------- | :------------------- | :---------------- | :--------- | :----------- | :-------------------
9999999 | 6966666 | 0.0.0.212 | 10.257.257.42 | b3c.16x64.encrypted | dal10 | Kube-0000067d | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | private | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-1e501db | default | 
9999999 | 6966666 | 0.0.0.195 | 10.257.257.82 | b3c.16x64.encrypted | dal10 | Kube-00001ab8 | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | private | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-1e501db | default | 
9999999 | 6966666 | 0.0.0.249 | 10.257.257.74 | b3c.16x64.encrypted | dal10 | Kube-00001c81 | normal | Ready | 2022-12-08T22:22:32+0000 |  |  |  | private | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-1e501db | default | 
9999999 | 6966666 | 0.0.0.242 | 10.257.257.85 | b3c.16x64.encrypted | dal10 | Kube-00001d6b | normal | Ready | 2022-12-08T22:22:32+0000 |  |  |  | private | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-1e501db | default | 
9999999 | 6966666 | 0.0.0.194 | 10.257.257.90 | b3c.16x64.encrypted | dal10 | Kube-00001ea3 | normal | Ready | 2022-12-08T22:22:29+0000 |  |  |  | private | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-1e501db | default | 
9999999 | 6966666 | 0.0.0.251 | 10.257.257.66 | b3c.8x32.encrypted | dal10 | Edge-00001fde | normal | Ready | 2022-12-08T22:22:32+0000 |  |  |  | public | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-de32bdc | edge | 
9999999 | 6966666 | 0.0.0.198 | 10.257.257.68 | b3c.8x32.encrypted | dal10 | Edge-00002009 | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | public | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-de32bdc | edge | 
9999999 | 6966666 | 0.0.0.207 | 10.257.257.94 | b3c.8x32.encrypted | dal10 | Edge-000021fa | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | public | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-de32bdc | edge | 



## Addon Status
OK
Name                  Version                Health State   Health Status   
addon1   1.0.0                  normal         Addon Ready.  
addon2                 1.14* (1.16 default)   normal        


---<br><br>


test

Now to get the replacement value:

test=Kube_cluster
replace2=$(awk  'BEGIN {p=0};
/'$test'/ {p=1 ;next};
/---<br><br>/ {p=0;next};
p {print }' "md/$test.md" )

If I attempt a replace I get regular expression errors:

awk '{gsub(/'$replace'/,&'$replace2'); print}'

errors with

   wk: illegal primary in regular expression  | privateVlan   | publicVlan   | privateIP   | publicIP   | machineType   | location   | id   | state   | status   | statusDate   | statusDetails   | errorMessage   | errorMessageDate   | isolation   | kubeVersion   | targetVersion   | reasonForDelete   | versionEOS   | masterVersionEOS   | trustedStatus   | poolid   | poolName   | pendingOperation    | at 
    source line number 1

Can a experienced Awk'r help me with what I'm doing incorrectly?

1st run of script you get the initial markdown (replace) ... Second run you get the same markdown file with the updated markdown. Where the entire first Initial markdown is replaced in the file with the replace2 markdown. Each run this file may contain other things in it... But the main goal is to search and find between tags and replace everything betweeen two tags with the new markdown table.

The expected result would be every time a run of the script is done... The Awk script would replace the text between the $test variable and -- . not replacing the $Test variable or the -- .

please update the question to clarify which part of the input you're referring to when you say *`I get different Markdown. That I want to replace`*; are you referring to the (entire) `|`-delimited table? something else? — markp-fuso, Dec 09 '22 at 19:13
while it's possible to pass `bash` variables to `awk`, the method you're using (piecing together separate `'`-delimited scripts with `bash` variable squeezed in between) is invalid if simply because your `bash` variable contains spaces and newline characters ... which leads `awk` to complain about invalid commands; you can see this if you enable `bash` debug mode (`set -xv`) and run your script; note: disable debug mode => `set +xv` — markp-fuso, Dec 09 '22 at 19:18
ok so if I want to use -v varname=value ... how do i put the awk var between these then /var/ ??? — Thom Schumacher, Dec 09 '22 at 19:20
replacing multiple lines typically comes down to loading the replacement lines into memory (eg, `awk` array) and then when processing the source file you skip the lines of interest and replace them with the contents of the array; this is likely doable in your case but we need a better explanation of what you're attempting; adding a section to the question that shows the expected output would also help (making sure it corresponds to sample inputs, ie, inputs and output match) — markp-fuso, Dec 09 '22 at 19:21
Yes replace contains the entire table and replace2 contains the same table with a few things that were updated. — Thom Schumacher, Dec 09 '22 at 19:21
Expected output would be a markdown file that contains the updated Table.. — Thom Schumacher, Dec 09 '22 at 19:22
*`I can find the Text i want to search for using this awk command`*; are you sure about that? run `typeset -p replace` to see what's actually stored in the variable — markp-fuso, Dec 09 '22 at 19:23
yes i did a typeset -p replace2 and i got the entire contents of replace2 — Thom Schumacher, Dec 09 '22 at 19:25
I would suggest you update the question with all of these clarifications; not everyone is going to read through comments trying to piece together the bigger picture; something along the lines of: here's my first set of input; here's my second set of input; objective is to replace the `|`-delimited table in the first set of input with the `|`-delimited table from the second of input, and this is the expected output: .... — markp-fuso, Dec 09 '22 at 19:25
the question is ... are the contents of `replace2` what you were expecting? your `awk` call is invalid, should be generating errors, with the net result being that what's stored in `replace2` is likely *not* what you're expecting — markp-fuso, Dec 09 '22 at 19:28
at the end of the day if I have any multiline file and i want to replace everything between two "markers" regardless of LF or CR or special chars.. what is the best means to do that? — Thom Schumacher, Dec 09 '22 at 19:30
read the 'new' set of lines into an `awk` array, then when processing the 'old' file you replace the 'old' lines with the 'new' lines (ie, contents of the array); doable but need you to update the question with better clarification on what is to be replaced; I originally asked if you want to replace the `|`-delimited table and you said yes but then you said you want to replace *everything* between `$test` and `--

` ... those aren't the same; which is it ... *just* the `|`-delimited table or *all* lines between the markers? — markp-fuso, Dec 09 '22 at 19:52
Read [how-do-i-use-shell-variables-in-an-awk-script](https://stackoverflow.com/questions/19075671/how-do-i-use-shell-variables-in-an-awk-script) to directly address `awk '{gsub(/'$replace'/,&'$replace2'); print}'`. — Ed Morton, Dec 10 '22 at 14:47

markp-fuso · Accepted Answer · 2022-12-09T22:08:29.947

Asusmptions/Understandings:

OP has two input files, we'll call them Kube_cluster.old.md and Kube_cluster.new.md
of interest in both files are the block of lines that start with # Kube_cluster and end with --- 
the objective is to replace this block of lines in the old file with the block of lines from the new file

For sake of visibility I've reduced the number of input lines as well as prefixed all lines (except the marker lines) with OLD or NEW:

$ cat Kub*old*
OLD test 1
OLD
# Kube_cluster
OLD| privateVlan   | publicVlan   | privateIP   | publicIP   | machineType   | location   | id   | state   | status   | statusDate   | statusDetails   | errorMessage   | errorMessageDate   | isolation   | kubeVersion   | targetVersion   | reasonForDelete   | versionEOS   | masterVersionEOS   | trustedStatus   | poolid   | poolName   | pendingOperation    |
OLD| --------------: | :------------- | :------------ | :----------- | :-------------- | :----------- | :----- | :-------- | :--------- | :------------- | :---------------- | :--------------- | :------------------- | :------------ | :-------------- | :---------------- | :------------------ | :------------- | :------------------- | :---------------- | :--------- | :----------- | :-------------------
OLD 9999999 | 6966666 | 0.0.0.212 | 10.257.257.42 | b3c.16x64.encrypted | dal10 | Kube-0000067d | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | private | 1.23.14_1553 | 1.23.14_1554 |  |  |  | unsupported | okiedokie-1e501db | default |
OLD 9999999 | 6966666 | 0.0.0.195 | 10.257.257.82 | b3c.16x64.encrypted | dal10 | Kube-00001ab8 | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | private | 1.23.14_1553 | 1.23.14_1554 |  |  |  | unsupported | okiedokie-1e501db | default |
OLD 9999999 | 6966666 | 0.0.0.249 | 10.257.257.74 | b3c.16x64.encrypted | dal10 | Kube-00001c81 | normal | Ready | 2022-12-08T22:22:32+0000 |  |  |  | private | 1.23.14_1553 | 1.23.14_1554 |  |  |  | unsupported | okiedokie-1e501db | default |
OLD
OLD ## Addon Status
OLD OK
OLD Name                  Version                Health State   Health Status
OLD addon1   1.0.0                  normal         Addon Ready.
OLD addon2                 1.14* (1.16 default)   normal
OLD
---<br><br>
OLD
OLD test 1

$ cat Kub*new*
NEW test 2
NEW
# Kube_cluster
NEW| privateVlan   | publicVlan   | privateIP   | publicIP   | machineType   | location   | id   | state   | status   | statusDate   | statusDetails   | errorMessage   | errorMessageDate   | isolation   | kubeVersion   | targetVersion   | reasonForDelete   | versionEOS   | masterVersionEOS   | trustedStatus   | poolid   | poolName   | pendingOperation    |
NEW| --------------: | :------------- | :------------ | :----------- | :-------------- | :----------- | :----- | :-------- | :--------- | :------------- | :---------------- | :--------------- | :------------------- | :------------ | :-------------- | :---------------- | :------------------ | :------------- | :------------------- | :---------------- | :--------- | :----------- | :-------------------
NEW 9999999 | 6966666 | 0.0.0.212 | 10.257.257.42 | b3c.16x64.encrypted | dal10 | Kube-0000067d | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | private | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-1e501db | default |
NEW 9999999 | 6966666 | 0.0.0.195 | 10.257.257.82 | b3c.16x64.encrypted | dal10 | Kube-00001ab8 | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | private | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-1e501db | default |
NEW 9999999 | 6966666 | 0.0.0.249 | 10.257.257.74 | b3c.16x64.encrypted | dal10 | Kube-00001c81 | normal | Ready | 2022-12-08T22:22:32+0000 |  |  |  | private | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-1e501db | default |
NEW
NEW ## Addon Status
NEW OK
NEW Name                  Version                Health State   Health Status
NEW addon1   1.0.0                  normal         Addon Ready.
NEW addon2                 1.14* (1.16 default)   normal
NEW
---<br><br>
NEW
NEW test 2

One awk idea:

marker='Kube_cluster'

awk -v marker="${marker}" '

# FNR==NR ==> 1st file processing:

FNR==NR { if ($0 ~ "--<br><br>")                   # end of block?
             collect=0                             # clear "collect"tion flag

          if (collect)                             # if collection flag is set then ...
             lines[++cnt]=$0                       # save current line

          if (index($0,"# " marker) == 1)          # start of block?
             collect=1                             # set collection flag
          next                                     # skip to next line of input
        }

# 2nd file processing:

        { if ($0 ~ "--<br><br>")                   # end of block?
             replace=0                             # disable replacement flag

          if (! replace)                           # replacement flag not set?
             print                                 # print current line

          if (index($0,"# " marker) == 1) {        # start of block?
             replace=1                             # set replacement flag
             for (i=1;i<=cnt;i++)                  # print saved block of lines from 1st file
                 print lines[i]
          }
        }
' "${marker}".new.md "${marker}".old.md

This generates:

OLD test 1
OLD
# Kube_cluster
NEW| privateVlan   | publicVlan   | privateIP   | publicIP   | machineType   | location   | id   | state   | status   | statusDate   | statusDetails   | errorMessage   | errorMessageDate   | isolation   | kubeVersion   | targetVersion   | reasonForDelete   | versionEOS   | masterVersionEOS   | trustedStatus   | poolid   | poolName   | pendingOperation    |
NEW| --------------: | :------------- | :------------ | :----------- | :-------------- | :----------- | :----- | :-------- | :--------- | :------------- | :---------------- | :--------------- | :------------------- | :------------ | :-------------- | :---------------- | :------------------ | :------------- | :------------------- | :---------------- | :--------- | :----------- | :-------------------
NEW 9999999 | 6966666 | 0.0.0.212 | 10.257.257.42 | b3c.16x64.encrypted | dal10 | Kube-0000067d | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | private | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-1e501db | default |
NEW 9999999 | 6966666 | 0.0.0.195 | 10.257.257.82 | b3c.16x64.encrypted | dal10 | Kube-00001ab8 | normal | Ready | 2022-12-08T22:22:31+0000 |  |  |  | private | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-1e501db | default |
NEW 9999999 | 6966666 | 0.0.0.249 | 10.257.257.74 | b3c.16x64.encrypted | dal10 | Kube-00001c81 | normal | Ready | 2022-12-08T22:22:32+0000 |  |  |  | private | 1.23.15_15563 | 1.23.15_15564 |  |  |  | unsupported | okiedokie-1e501db | default |
NEW
NEW ## Addon Status
NEW OK
NEW Name                  Version                Health State   Health Status
NEW addon1   1.0.0                  normal         Addon Ready.
NEW addon2                 1.14* (1.16 default)   normal
NEW
---<br><br>
OLD
OLD test 1

Thank you mark for the extra Comments in your answer and forcing me to SPELL out the requirements Exactly — Thom Schumacher, Dec 09 '22 at 22:35
hmm i mark old and new as you did the only one I got with new is the ----

line — Thom Schumacher, Dec 09 '22 at 23:39
I removed the ! from if (! replace) and now I get the reverse.. Which is what you have I think ... actually this actually worked... changing the FNR=NR to the reverse.. then I get the new file — Thom Schumacher, Dec 09 '22 at 23:50
@ThomSchumacher what order did you feed the files to the `awk` script? as coded you have to supply first the 'new' file and then the 'old' file — markp-fuso, Dec 10 '22 at 00:25

Replacing multi line with another multi line using AWK

1 Answers1