How to find the nth multiline block of text using sed

Question

So I have a file which contains blocks that looks as follows:

menuentry ... {
....
....
}

....

menuentry ... {
....
....
}

I need to look at the contents of each menu entry in a bash script. I have very limited experience with sed but through a very exhaustive search I was able to construct the following:

cat $file | sed '/^menuentry.*{/!d;x;s/^/x/;/x\{1\}/!{x;d};x;:a;n;/^}/!ba;q'

and I can replace the \{1\} with whatever number I want to get the nth block. This works fine like that, but the problem is I need to iterate through an arbitrary number of times:

numEntries=$( egrep -c "^menuentry.*{" $file ) 
for i in $( seq 1 $numEntries); do

    i=$( echo $i | tr -d '\n' ) # A google search indicated sed  encounters problems
                                # when a substituted variable has a trailing return char

    # Get the nth entry block of text with the sed statement from before, 
    # but replace with variable $i
    entry=$( cat $file | sed '/^menuentry.*{/!d;x;s/^/x/;/x\{$i\}/!{x;d};x;:a;n;/^}/!ba;q')

    # Do some stuff with $entry #

done

I've tried with every combination of quotes/double quotes and braces around the variable and around the sed statement and every which way I do it, I get some sort of error. As I said, I don't really know much about sed and that frankenstein of a statement is just what I managed to mish-mosh together from various google searches, so any help or explanation would be much appreciated!

TIA

Where this file comings from ? Is it a conf file ? If yes, which tool ? — Gilles Quénot, Feb 23 '18 at 16:17
So you need to have contents from `menuentry ... {` to `}` in different files or a single output file? — RavinderSingh13, Feb 23 '18 at 16:17
Yes, I believe it is a boot config file, but I am not making any changes to the file, I just need to grab some info from it. — WarBro, Feb 23 '18 at 16:20
I don't need the contents from `menuentry ... { to }` in any outputfiles, just stored in the variable $entry for a bit — WarBro, Feb 23 '18 at 16:21
There's some grub parser around: https://metacpan.org/pod/Linux::Bootloader::Grub — Gilles Quénot, Feb 23 '18 at 16:32
Unfortunately, I cannot use any external tools -- that's one of the reasons I'm not doing this with say python or even perl. My expected output is on each iteration $entry should contain "`menuentry ... { to }" which I will then inspect the contents of which need to match some minimum requirements, and if not, I will take note of which menuentry is not configured properly. — WarBro, Feb 23 '18 at 17:33
How about splitting this work up? First search for lines, where a menu entry starts: `entrystarts=($(sed -n '/^menuentry.*/=' /boot/grub/grub.cfg))`, then, in a second step, choose a starting val from the array, for the n-th entry ${entrystarts[$n]} and proceed from there? — user unknown, Feb 23 '18 at 20:56
Not even going to pretend I understand the sed command but I understand the part of bash that was failing. Awesome sed command by the way I'll probably pick your brain in the future. — penguinjeff, Feb 23 '18 at 22:19
sorry to confess, that i am interested in your problem, except that i am not interested in reading your sed command. if you could, summarizing what you want to do in english will help more people to understand what you are trying to do. — Jason Hu, Feb 24 '18 at 21:08

Ed Morton · Accepted Answer · 2018-02-24T19:30:00.857

sed is for simple substitutions on individual lines, that is all, and shell loops to manipulate text are immensely slow and difficult to write robustly. The folks who invented sed and shell also invented awk for tasks like this:

awk -v RS= 'NR==3' file

would print the 3rd blank-line-separated block of text as shown in your question. This would print every block containing the string "foobar":

awk -v RS= '/foobar/' file

and anything else you might want to do is equally trivial.

The above will work efficiently, robustly and portably using any awk in any shell on any UNIX box. For example:

$ cat file
menuentry first {
    Now is the Winter
    of our discontent
}

menuentry second {
    Wee sleekit cowrin
    timrous beastie,
    oh whit a panic's in
    thy breastie.
}

menuentry third {
    Twas the best of times
    Twas the worst of times
    Make up your damn mind
}

.

$ awk -v RS= 'NR==3' file
menuentry third {
    Twas the best of times
    Twas the worst of times
    Make up your damn mind
}

$ awk -v RS= 'NR==2' file
menuentry second {
    Wee sleekit cowrin
    timrous beastie,
    oh whit a panic's in
    thy breastie.
}

$ awk -v RS= '/beastie/' file
menuentry second {
    Wee sleekit cowrin
    timrous beastie,
    oh whit a panic's in
    they breastie.
}

If you find yourself trying to do anything other than s/old/new with sed and/or using sed commands other than s, g and p (with -n) then you are using constructs that became obsolete in the mid-1970s when awk was invented.

If the above doesn't work for you then edit your question to provide more truly representative sample input and expected output.

penguinjeff · Answer 2 · 2018-02-26T16:54:02.033

0

I see the problem. You are trying to use $i inside single quotes. $i will be left as $i As I posted below correcting it with single quotes around $i so that bash will see it as 3 strings

'one string'"$twostrings"'three strings'

#!/bin/bash
#filename:grubtest.sh

numEntries=$( egrep -c "^menuentry.*{" "$1" )
i=0 
while [ $i -lt $numEntries ]; do
    i=$(($i+1))
    # Get the nth entry block of text with the sed statement from before, 
    entry=$( cat "$1" | sed '/^menuentry.*{/!d;x;s/^/x/;/x\{'"$i"'\}/!{x;d};x;:a;n;/^}/!ba;q')

    # Do some stuff with $entry #
    echo $entry|cut -d\' -f2

done

this returned my grub items when ran like so ./grubtest.sh /etc/grub2.cfg

edited Feb 26 '18 at 16:54

answered Feb 23 '18 at 18:30

penguinjeff

306
3
8

As I said, I tried with every combination of quotes and double quotes, both around $i and around the whole sed statement. For some reason, all variants generate an error. One of the errors is something like "unterminted address regex" and another one is "invalid content of \{\}" – WarBro Feb 23 '18 at 19:01
here you go that seems to be happy sed '/^menuentry.*{/!d;x;s/^/x/;/x\{'$i'\}/!{x;d};x;:a;n;/^}/!ba;q' – penguinjeff Feb 23 '18 at 20:23
just wrap single quotes around $i so that it is 3 different strings one single quote string one interpreted $i string and another single quote string – penguinjeff Feb 23 '18 at 20:31
`i=$( echo $i | tr -d '\n' )` does nothing since `$i` cannot contain a newline and you've got to quote your variables everywhere - `"$foo"`, not `$foo`. The line containing sed command, for example, with appropriate quoting would be `entry=$( cat "$1" | sed '/^menuentry.*{/!d;x;s/^/x/;/x\{'"$i"'\}/!{x;d};x;:a;n;/^}/!ba;q')` but of course you should never actually use this approach anyway as it'll be immensely inefficient and error-prone. – Ed Morton Feb 24 '18 at 19:23
Starting fresh I would have used a different method to loop. I just changed the OP's $file to $1 added single quotes around $i and added something to see that it was working. I did very little editing of his code. His code seemed to work for me. Other than what I had stated. ;-) – penguinjeff Feb 24 '18 at 20:35
You only have to quote a variable if it has spaces in it that you want. Since $i it is just a number the quotes don't make a difference. You are right about the quotes around $1 or $filename in the case of the OP. – penguinjeff Feb 24 '18 at 20:58
Changed to a looping method I am more familiar with. – penguinjeff Feb 24 '18 at 21:12
`You only have to quote a variable if ...` - no. The correct statement starts with `You only have to **not** quote a variable if...`. Quoting variables isn't something you do when it proves necessary, it's what you **always** do unless you have a very specific purpose in mind that you have to remove the quotes to accomplish and fully understand all of the consequences (and those include more than just word splitting at spaces). With `$i`, for example, you're not trying to do anything that would require you to leave it unquoted so you should not leave it unquoted. – Ed Morton Feb 25 '18 at 02:00
Well thank you Ed Morton. I will go back through my years of scripting and quote the hell out of them. You are probably correct. I had no formal training. I don't think I left variables unquoted too often. However with the case above quoting in the sed section makes sense but the comparison like I used in the while I could have sworn I had it complain to me using -lt or -gt or any number comparison using quotes around the number. – penguinjeff Feb 26 '18 at 02:29

score 0 · Answer 3 · answered Feb 23 '18 at 21:05

How about splitting this work up?

First search for lines, where a menu entry starts (for personal reasons, I here use grub, not grub2, adjust to your needs):

entrystarts=($(sed -n '/^menuentry.*/=' /boot/grub/grub.cfg))

Then, in a second step, choose a starting val from the array, for the n-th entry ${entrystarts[$n]} and proceed from there? Afaik, the entry end is easy to detect by a single closing curly brace.

for i in ${entrystarts[@]}
do 
    // your code here, proof of concept (note grub/grub2):
    sed -n "$i,/}/p" /boot/grub/grub.cfg
done

How to find the nth multiline block of text using sed

3 Answers3