sed : Select lines between two patterns

Question

Given a text file containing the following:

1
2
REGEX1
3    - multiple line block
4
REGEX2
5
6
REGEX1
7    - multiple line block
REGEX2
8
9
REGEX1
10    - multiple line block

I want to extract the following:

REGEX1
3    - multiple line block
4
REGEX1
7    - multiple line block
REGEX1
10    - multiple line block

i.e I want to extract lines including between REGEX1 and REGEX2 including REGEX1, but not REGEX2.

I have written a sed script : sed -n '/REGEX1/,/REGEX2/{/REGEX2/!p}' file.

It works fine, but when given an instance like this:

1
2
REGEX2 REGEX1
3    - multiple line block
4
REGEX2
5
6
REGEX2 REGEX1
7    - multiple line block
REGEX2
8
9
REGEX2 REGEX1
10    - multiple line block

My script only gives me:

3     - multiple line block
4
7     - multiple line block
10    - multiple line block

Where as I want it to output to be:

REGEX2 REGEX1
3    - multiple line block
4
REGEX2 REGEX1
7    - multiple line block
REGEX2 REGEX1
10    - multiple line block

How can I achieve this, without inefficiency (like storing line numbers and going through the file once again) ?

what is the expected output for second sample? perhaps you are looking for line anchors? `sed -n '/REGEX1/,/REGEX2/{/^REGEX2$/!p}'` — Sundeep, Jun 20 '17 at 10:46
well then `sed -n '/REGEX1/,/REGEX2/{/^REGEX2$/!p}'` is what you are looking for? — Sundeep, Jun 20 '17 at 11:14

Ed Morton · Accepted Answer · 2017-06-20T14:12:35.127

Strike 1: sed is for simple substitutions on individual lines, that is all. Anything else requires constructs other than s, g, and p (with -n) and those all became obsolete in the mid-1970s when awk was invented.

Strike 2: you should never use range expressions as they make trivial tasks very slightly briefer but then require a complete rewrite or duplicate conditions when the task becomes the slightest bit more interesting, use flag variables instead.

Strike 3: sed doesn't support variables so you can't use flags to tell when you're in/out of the blocks of text you care about.

So - just use awk:

$ awk '/REGEX2/{f=0} /REGEX1/{f=1} f' file
REGEX1
3    - multiple line block
4
REGEX1
7    - multiple line block
REGEX1
10    - multiple line block

and on your second set of input:

$ awk '/REGEX2/{f=0} /REGEX1/{f=1} f' file
REGEX2 REGEX1
3    - multiple line block
4
REGEX2 REGEX1
7    - multiple line block
REGEX2 REGEX1
10    - multiple line block

The above will work robustly and efficiently on any size file with any awk on any UNIX box.

See https://stackoverflow.com/a/17914105/1745001 for more ways to select blocks of text.

Yes. Hopefully the fact that it's extremely obvious what the awk solution is doing whereas you had to ask for an explanation of the sed solution isn't lost on anyone! — Ed Morton, Jun 20 '17 at 14:09

potong · Answer 2 · 2017-06-20T17:03:49.677

0

This might work for you (GNU sed):

sed -r '/^REGEX/h;G;s/^.*((REGEX1\b).*\n.*\2)/\1/;/\n.*REGEX1\b/P;d' file

Store the REGEX in the hold space and append it to following records. If the regexp is matched in the appended part of the line print the first half otherwise delete the line.

EDIT:

Change to the original question; the following simpler solution satisfies:

sed '/^REGEX1/{:a;n;/REGEX2/!ba};d' file

However if the REGEX2 REGEX1 repeat this needs to be altered to:

sed ':a;/^REGEX1/{:b;n;/REGEX2/!bb;ba};d' file

edited Jun 20 '17 at 17:03

answered Jun 20 '17 at 10:56

potong

55,640
6
51
83

I'm sorry I didn't mean the same output. I have edited my question to reflect the same. But I'm quite intrigued as to how the above command works. Care to explain? – gitmorty Jun 20 '17 at 14:06

score 0 · Answer 3 · answered Jun 20 '17 at 14:28

0

You can just bandaid your original sed a little more.

sed -n '/REGEX1/,/REGEX2/{/REGEX1/{p;n};/REGEX2/!p}' file

Adding /REGEX1/{p;n} ensures that REGEX1 lines print, and then n immediately replaces the content of the pattern space with the next line.

I don't like how you have to repeat yourself with /START/,/END/ when there are special cases for /START/ and /END/ either, but it does seem like you can stick with sed here by just using n judiciously.

n will burn you, though, if you had subsequent sed commands. You could pipe to another sed invocation... or use awk.

answered Jun 20 '17 at 14:28

stevesliva

5,351
1
16
39

Are you sure this works? My `sed` complains sed: 1: `'/REGEX1/,/REGEX2/{/REGEX1/{p; ... : extra characters at the end of n command.` – gitmorty Jun 21 '17 at 05:14
I'm currently on OSX, so that might be the problem. The command works fine on GNU `sed`. But It'd be nice to know how to make it work with the BSD `sed` too. – gitmorty Jun 21 '17 at 05:21
@AkhilAvinash OSX/BSD sed just wants a semicolon after the last command inside the `{}` – stevesliva Jun 21 '17 at 14:10

sed : Select lines between two patterns

3 Answers3