How to select lines between two marker patterns which may occur multiple times with awk/sed

Question

Using awk or sed how can I select lines which are occurring between two different marker patterns? There may be multiple sections marked with these patterns.

For example: Suppose the file contains:

abc
def1
ghi1
jkl1
mno
abc
def2
ghi2
jkl2
mno
pqr
stu

And the starting pattern is abc and ending pattern is mno So, I need the output as:

def1
ghi1
jkl1
def2
ghi2
jkl2

I am using sed to match the pattern once:

sed -e '1,/abc/d' -e '/mno/,$d' <FILE>

Is there any way in sed or awk to do it repeatedly until the end of file?

score 242 · Accepted Answer · edited May 23 '17 at 12:02

242

Use awk with a flag to trigger the print when necessary:

$ awk '/abc/{flag=1;next}/mno/{flag=0}flag' file
def1
ghi1
jkl1
def2
ghi2
jkl2

How does this work?

/abc/ matches lines having this text, as well as /mno/ does.
/abc/{flag=1;next} sets the flag when the text abc is found. Then, it skips the line.
/mno/{flag=0} unsets the flag when the text mno is found.
The final flag is a pattern with the default action, which is to print $0: if flag is equal 1 the line is printed.

For a more detailed description and examples, together with cases when the patterns are either shown or not, see How to select lines between two patterns?.

edited May 23 '17 at 12:02

Community

1
1

answered Aug 01 '13 at 08:29

fedorqui

275,237
103
548
598

44

If you want to print everything between and *including* the pattern then you can use `awk '/abc/{a=1}/mno/{print;a=0}a' file`. – scai Nov 07 '13 at 08:08
8

Yes, @scai ! or even `awk '/abc/{a=1} a; /mno/{a=0}' file` - with this, putting `a` condition before the `/mno/` we make it evaluate the line as true (and print it) before setting `a=0`. This way we can avoid writing `print`. – fedorqui Nov 07 '13 at 09:43
19

@scai @fedorqui For including pattern output, you can do `awk '/abc/,/mno/' file` – Jotne Dec 04 '13 at 06:44
@scai,@fedorqui : In the above how can we include abc and exclude mno. – hkasera Dec 10 '14 at 18:04
1

@hkasera `awk '/abc/{flag=1}/mno/{flag=0}flag' file` should make. – fedorqui Dec 11 '14 at 08:54
1

The only drawback of this solution if patterns `abc` and `mno` are the same – Eir Nym Apr 22 '17 at 13:32
2

@EirNym that is a weird scenario that can be handled on very different ways: which lines would you like to print? Probably `awk 'flag; /PAT1/{flag=1; next} /PAT1/{flag=0}' file` would make. – fedorqui Apr 24 '17 at 08:28
@EirNym Have a look at `seq 100 | awk '/.*1$/{ flag=1-flag }flag'` so see a way of doing it. – Alfe Nov 22 '17 at 13:37
2

Hi @fedorqui, how to stop after first occurance ? – Brain90 Dec 13 '17 at 08:06
@Brain90 probably `/second_pattern/ {exit}` should make. – fedorqui Dec 13 '17 at 23:17
Hi @fedorqui I have a doubt . If abc pattern is repeated twice before mno as follows . `abc def1 abc ghi1 jkl1 mno abc def2 ghi2 jkl2 mno pqr stu ` .. Here the command will print lines between abc and mno including abc.How to avoid this and get the result for words between abc and mno only. – Eashwar Gadanchi Jan 19 '18 at 10:20
@EashwarGadanchi the approach in my answer keeps printing while a true condition is happening. To get what you mention we would have to create some kind of buffering, since it needs to into account things that are not known when you are reading a specific line. See [other approaches to this problem](https://stackoverflow.com/q/38972736/1983854) in a related question I created. – fedorqui Jan 23 '18 at 10:50
3

For newbies like me, there is a [doc](https://www.gnu.org/software/gawk/manual/html_node/Action-Overview.html). 1. A awk "rule" contains a "pattern" and an "action", either of which (but not both) may be omitted. So `[pattern] { action }` or `pattern [{ action }]`. 2. An action consists of one or more awk statements, enclosed in braces (‘{…}’). —— So the ending `flag` is abbr of `flag {print $0}` – Weekend Jan 07 '21 at 08:40

score 54 · Answer 2 · answered Aug 01 '13 at 08:47

54

Using sed:

sed -n -e '/^abc$/,/^mno$/{ /^abc$/d; /^mno$/d; p; }'

The -n option means do not print by default.

The pattern looks for lines containing just abc to just mno, and then executes the actions in the { ... }. The first action deletes the abc line; the second the mno line; and the p prints the remaining lines. You can relax the regexes as required. Any lines outside the range of abc..mno are simply not printed.

answered Aug 01 '13 at 08:47

Jonathan Leffler

730,956
141
904
1,278

1

@JonathanLeffler can I know what is the purpose of using `-e` – Kasun Siyambalapitiya Dec 06 '16 at 04:33
1

@KasunSiyambalapitiya: Mostly it means I like to use it. Formally, it specifies that the next argument is (part of) the script that `sed` should execute. If you want or need to use several arguments to include the entire script, then you must use `-e` before each such argument; otherwise, it's optional (but explicit). – Jonathan Leffler Dec 06 '16 at 04:41
Nice! (I prefer sed over awk.) When using complex regular expressions, it would be nice not to have to repeat them. Isn't it possible to delete the first / last line of the "selected" range? Or to first apply the ``d`` to all lines up to the first match, and then another ``d`` to all lines starting with the second match? – hans_meine Dec 08 '16 at 10:12
(Replying to my own comment.) If there's only one section to be cut, I could tentatively solve this e.g. for LaTeX using ``sed -n '1,/\\begin{document}/d;/\\end{document}/d;p'``. (This is cheating a little bit, since the second part does not delete up to the document end, and I would not know how to cut multiple parts as the OP asked for.) – hans_meine Dec 08 '16 at 10:50
@JonathanLeffler what is the reason for inserting the `$` mark, as in `/^abc$` and others – Kasun Siyambalapitiya Jan 25 '17 at 04:58
The `^` matches the start of a line; the `$` matches the end of line. Between them, they ensure that only lines containing just `abc` and `mno` are matched. – Jonathan Leffler Jun 27 '17 at 12:23

score 20 · Answer 3 · answered Aug 01 '13 at 09:39

20

This might work for you (GNU sed):

sed '/^abc$/,/^mno$/{//!b};d' file

Delete all lines except for those between lines starting abc and mno

answered Aug 01 '13 at 09:39

potong

55,640
6
51
83

`!d;//d` golfs 2 characters better :-) http://stackoverflow.com/a/31380266/895245 – Ciro Santilli OurBigBook.com Jul 13 '15 at 09:54
1

This is awesome. The `{//!b}` prevents the `abc` and `mno` from being included in the output, but I can't figure out how. Could you explain? – Brendan Feb 16 '17 at 17:44
2

@Brendan the instruction `//!b` reads if the current line is neither one of the lines that match the range, break and therefore print those lines otherwise all other lines are deleted. – potong Feb 17 '17 at 01:14

score 15 · Answer 4 · edited May 23 '17 at 12:26

15

sed '/^abc$/,/^mno$/!d;//d' file

golfs two characters better than ppotong's {//!b};d

The empty forward slashes // mean: "reuse the last regular expression used". and the command does the same as the more understandable:

sed '/^abc$/,/^mno$/!d;/^abc$/d;/^mno$/d' file

This seems to be POSIX:

If an RE is empty (that is, no pattern is specified) sed shall behave as if the last RE used in the last command applied (either as an address or as part of a substitute command) was specified.

edited May 23 '17 at 12:26

Community

1
1

answered Jul 13 '15 at 09:53

Ciro Santilli OurBigBook.com

347,512
102
1,199
985

1

I think the second solution will end up with nothing as the second command is also a range. However kudos for the first. – potong Jul 13 '15 at 14:20
@potong true! I have to study more why the first one works. Thanks! – Ciro Santilli OurBigBook.com Jul 13 '15 at 14:22

score 14 · Answer 5 · edited Jan 02 '20 at 05:53

14

From the previous response's links, the one that did it for me, running ksh on Solaris, was this:

sed '1,/firstmatch/d;/secondmatch/,$d'

1,/firstmatch/d: from line 1 until the first time you find firstmatch, delete.
/secondmatch/,$d: from the first occurrance of secondmatch until the end of file, delete.
Semicolon separates the two commands, which are executed in sequence.

edited Jan 02 '20 at 05:53

Irfan Latif

498
2
9
24

answered Jul 12 '17 at 16:38

FanDeLaU

318
2
8

Just curious, why does the range limiter (`1,`) come before `/firstmatch/`? I'm guessing this could also be phrased `'/firstmatch/1,d;/secondmatch,$d'`? – Luke Davis Jun 25 '18 at 00:40
3

With "1,/firstmatch/d" you are saying "from line 1 until the first time you find 'firstmatch', delete". Whereas, with "/secondmatch/,$d" you say "from the first occurrance of 'secondmatch' until the end of file, delete". the semicolon separates the two commands, which are executed in sequence. – FanDeLaU Dec 20 '18 at 17:18

pataluc · Answer 6 · 2014-06-11T11:32:58.477

3

something like this works for me:

file.awk:

BEGIN {
    record=0
}

/^abc$/ {
    record=1
}

/^mno$/ {
    record=0;
    print "s="s;
    s=""
}

!/^abc|mno$/ {
    if (record==1) {
        s = s"\n"$0
    }   
}

using: awk -f file.awk data...

edit: O_o fedorqui solution is way better/prettier than mine.

edited Jun 11 '14 at 11:32

answered Aug 01 '13 at 08:44

pataluc

569
4
19

3

In GNU awk `if (record=1)` should be `if (record==1)`, i.e. double `=` - see [gawk comparison operators](https://www.gnu.org/software/gawk/manual/html_node/Comparison-Operators.html) – George Hawkins May 26 '14 at 08:53

score 3 · Answer 7 · edited Apr 13 '17 at 12:36

3

Don_crissti's answer from Show only text between 2 matching pattern?

firstmatch="abc"
secondmatch="cdf"
sed "/$firstmatch/,/$secondmatch/!d;//d" infile

which is much more efficient than AWK's application, see here.

edited Apr 13 '17 at 12:36

Community

1
1

answered Sep 11 '15 at 14:21

Léo Léopold Hertz 준영

134,464
179
445
697

I don't think linking the time comparisons makes much sense here, since the requirements of the questions are quite different, hence the solutions. – fedorqui Sep 11 '15 at 15:11
2

I disagree because we should have some criterias to compare answers. Only a few has SED applications. – Léo Léopold Hertz 준영 Sep 11 '15 at 16:10

score 2 · Answer 8 · answered Aug 01 '13 at 09:13

2

perl -lne 'print if((/abc/../mno/) && !(/abc/||/mno/))' your_file

answered Aug 01 '13 at 09:13

Vijay

65,327
90
227
319

Good to know perl equivalent as it is a pretty good alternative to both awk and sed. – akhan Mar 08 '17 at 23:46

Weekend · Answer 9 · 2019-01-02T09:14:01.387

I tried to use awk to print lines between two patterns while pattern2 also match pattern1. And the pattern1 line should also be printed.

e.g. source

package AAA
aaa
bbb
ccc
package BBB
ddd
eee
package CCC
fff
ggg
hhh
iii
package DDD
jjj

should has an ouput of

package BBB
ddd
eee

Where pattern1 is package BBB, pattern2 is package \w*. Note that CCC isn't a known value so can't be literally matched.

In this case, neither @scai 's awk '/abc/{a=1}/mno/{print;a=0}a' file nor @fedorqui 's awk '/abc/{a=1} a; /mno/{a=0}' file works for me.

Finally, I managed to solve it by awk '/package BBB/{flag=1;print;next}/package \w*/{flag=0}flag' file, haha

A little more effort result in awk '/package BBB/{flag=1;print;next}flag;/package \w*/{flag=0}' file, to print pattern2 line also, that is,

package BBB
ddd
eee
package CCC

blhsing · Answer 10 · 2021-03-05T20:56:07.060

0

This can also be done with logical operations and increment/decrement operations on a flag:

awk '/mno/&&--f||f||/abc/&&f++' file

edited Mar 05 '21 at 20:56

answered Mar 05 '21 at 20:50

blhsing

91,368
6
71
106

I'm absolutely certain that i've used awk in the past for this problem, and it was nothing like this complex. – Owl Mar 28 '22 at 10:45
1

Obviously the accepted answer in awk that predates my answer by more than 7 years is much more readable, and I saw that answer before I posted mine. I'm just throwing this one here because it is one byte shorter than the accepted answer even after renaming its variable `flag` to `f`, in the spirit of some good ol' code golf fun. :-) – blhsing Mar 30 '22 at 07:14

How to select lines between two marker patterns which may occur multiple times with awk/sed

10 Answers10

Linked

Related