Process a line based on lines before and after in bash

Question

I am trying to figure out how to write a bash script which uses the lines immediately before and after a line as a condition. I will give an example in a python-like pseudocode which makes sense to me.

Basically:

for line in FILE:
    if line_minus_1 == line_plus_one:
        line = line_minus_1

What would be the best way to do this?

So if I have an input file that reads:

3
1
1
1
2
2
1
2
1
1
1
2
2
1
2

my output would be:

3
1
1
1
2
2
2
2
1
1
1
2
2
2
2

Notice that it starts from the first line until the last line and respects changes made in earlier lines so if I have:

2
1
2
1
2
2

I would get:

2
2
2
2
2
2

and not:

2
1
1
1
2
2

Answer 1

$ awk 'minus2==$0{minus1=$0} NR>1{print minus1} {minus2=minus1; minus1=$0} END{print minus1}' file
3
1
1
1
2
2
2
2
1
1
1
2
2
2
2

How it works

• minus2==$0{minus1=$0}

  If the line from 2 lines ago is the same as the current line, then set the line from 1 line ago equal to the current line.

• NR>1{print minus1}

  If we are past the first line, then print the line from 1 line ago.

• minus2=minus1; minus1=$0

  Update the variables.

• END{print minus1}

  After we have finished reading the file, print the last line.

Multiple line version

For those who like their code spread over multiple lines:

awk '
    minus2==$0{
        minus1=$0
    }

    NR>1{
        print minus1
    }

    {
        minus2=minus1
        minus1=$0
    }

    END{
        print minus1
    }
    ' file

Answer 2

Here is a (GNU) sed solution:

$ sed -r '1N;N;/^(.*)\n.*\n\1$/s/^(.*\n).*\n/\1\1/;P;D' infile
3
1
1
1
2
2
2
2
1
1
1
2
2
2
2

This works with a moving three line window. A bit more readable:

sed -r '        # -r for extended regular expressions: () instead of \(\)
    1N          # On first line, append second line to pattern space
    N           # On all lines, append third line to pattern space
    /^(.*)\n.*\n\1$/s/^(.*\n).*\n/\1\1/    # See below
    P           # Print first line of pattern space
    D           # Delete first line of pattern space
' infile

N;P;D is the idiomatic way to get a moving two line window: append a line, print first line, delete first line of pattern space. To get a moving three line window, we read an additional line, but only once, namely when processing the first line (1N).

The complicated bit is checking if the first and third line of the pattern space are identical, and if they are, replacing the second line with the first line. To check if we have to make the substitution, we use the address

/^(.*)\n.*\n\1$/

The anchors ^ and $ are not really required as we'll always have exactly to newlines in the pattern space, but it makes it more clear that we want to match the complete pattern space. We put the first line into a capture group and see if it is repeated on the third line by using a backreference.

Then, if this is the case, we perform the substitution

s/^(.*\n).*\n/\1\1/

This captures the first line including the newline, matches the second line including the newline, and substitutes with twice the first line. P and D then print and remove the first line.

When reaching the end, the whole pattern space is printed so we're not swallowing any lines.

This also works with the second input example:

$ sed -r '1N;N;/^(.*)\n.*\n\1$/s/^(.*\n).*\n/\1\1/;P;D' infile2
2
2
2
2
2
2

---

To use with BSD sed (as found in OS X), you'd either have to use the -E instead of the -r option, or use no option, i.e., basic regular expressions and escape all parentheses (\(\)) in the capture groups. The newline matching should work, but I didn't test it. If in doubt, check this great answer lining out all the differences.