How do I filter out multiple lines which starts and ends with a given pattern?

Question

In my company, we have a huge log file with java stacktraces. In general, its format is:

useful line 1
useful line 2
useful line 3
MARKER1 start of exception
... <--Around 100 lines here
end of exception MARKER2
useful line 4
useful line 5
useful line 6
MARKER1 start of exception
... <--Around 100 lines here
end of exception MARKER2
useful line 7

It has useful information mixed with useless exceptions.

Is it possible to filter out the entire contents of useless exceptions from the logs using a combination of awk/sed/grep..?

In the example above, the output would be:

useful line 1
useful line 2
useful line 3
useful line 4
useful line 5
useful line 6
useful line 7

Thanks.

http://stackoverflow.com/questions/6287755/using-sed-to-delete-all-lines-between-two-matching-patterns — Ipor Sircer, Nov 05 '16 at 00:27
Please add sample input and your desired output for that sample input to your question. — Cyrus, Nov 05 '16 at 00:34
@Cyrus, I added sample input and output to my original post. — user674669, Nov 05 '16 at 01:06
see also https://stackoverflow.com/questions/38972736/how-to-select-lines-between-two-patterns — Sundeep, Nov 05 '16 at 02:15
@user674669: Your example ist useless. `grep "useful line" file` — Cyrus, Nov 05 '16 at 12:56

score 3 · Answer 1 · answered Nov 05 '16 at 01:15

Using awk

To exclude the start and end of exceptions and everything in between:

$ awk '/start of exception/,/end of exception/{next} 1' file
useful line 1
useful line 2
useful line 3
useful line 4
useful line 5
useful line 6
useful line 7

How it works:

/start of exception/,/end of exception/{next}

For any line in the range from the start to the end of the exception, we skip the rest of the commands and start over on the next line.
1

For any other lines, we print them. 1 is awk's shorthand for print-the-line.

Using sed

$ sed '/start of exception/,/end of exception/d' file
useful line 1
useful line 2
useful line 3
useful line 4
useful line 5
useful line 6
useful line 7

How it works:

/start of exception/,/end of exception/d

For any line in the range from the start to the end of the exception, we delete the line (d).

All other lines are, by default, printed.

score 3 · Answer 2 · answered Nov 05 '16 at 01:21

3

another sed with anchored patterns

$ sed '/^MARKER1/,/MARKER2$/d' file

useful line 1
useful line 2
useful line 3
useful line 4
useful line 5
useful line 6
useful line 7

or translated to awk

$ awk '/^MARKER1/,/MARKER2$/{next} 1' file

answered Nov 05 '16 at 01:21

karakfa

66,216
7
41
56

dawg · Answer 3 · 2016-11-05T01:56:34.153

2

Given your input, you can do:

$ awk 'BEGIN{ flag=1 } /MARKER/ {flag=!flag; next} flag' file
useful line 1
useful line 2
useful line 3
useful line 4
useful line 5
useful line 6
useful line 7

As pointed out in comments, you can also do:

awk '/MARKER/{f=!f;next} !f' file

edited Nov 05 '16 at 01:56

answered Nov 05 '16 at 01:15

dawg

98,345
23
131
206

1

perhaps shorter with `awk '/MARKER/{f=!f;next} !f'` – karakfa Nov 05 '16 at 01:51

score 0 · Answer 4 · answered Nov 05 '16 at 10:04

0

I think

cat filename | grep useful

will work

answered Nov 05 '16 at 10:04

Alessandro Blasetti

229
1
11

`grep useful filename` will work for the given example. The actual logfile might have other text, like starting with a timestamp for every useful line. In that case you can change your command into `grep "^2" logfile` of `grep -E " ERROR| WARN| INFO" logfile` when these lines are useful. – Walter A Nov 05 '16 at 10:19

How do I filter out multiple lines which starts and ends with a given pattern?

4 Answers4

Using awk

Using sed