Is it possible to check if a line is included in the next line in unix and if so, delete the next line

Question

Is it possible in a while loop, on a small file, to save the current line to a variable and check if that variable is included in the next line, if so, delete the next line;

Something like:

while read p ; do
   a gets value of p
   if a included in next line of file, delete next line
done<file

So if we have the lines:

abc man
abc man 13e1312 22
abc 12345
dec asdasfa sadfasfa
dec 22
dec 2232
dec 22 das fss

It will print:

abc man
abc 12345
dec asdasfa sadfasfa
dec 22

i need to check the whole line, not just until the first space, i edited the question for a more clear example — StressedStudent, Aug 22 '17 at 07:46
Again, show us what you have tried and explain what is not working and we'll gladly assist. Right now you just want as ready solution without showing any effort from your part and that is frowned upon heron SO. — Fredrik Pihl, Aug 22 '17 at 07:50
Of course yes but you need more logical code. For example well control tokenize of linereader, etc.... — Bui Anh Tuan, Aug 22 '17 at 08:06
Why isnt `dec 22 das fss` printed, `dec 2232` doesn't appear in the line. — 123, Aug 22 '17 at 08:51
@123: Because of `dec 22`. It's not necessarily the next line if I got the OP right. I understand it that way that it remebers the line if it doesn't match. The OP should clarify that. — Andre Kampling, Aug 22 '17 at 08:57
@AndreKampling ah right, could just do `awk '{for(i in a)if(index($0,i))next;print;a[$0]}'` — 123, Aug 22 '17 at 09:10

score 3 · Accepted Answer · edited Oct 23 '20 at 17:03

Here is a simple script that uses a while loop to read the file line per line and the string contains syntax. It will look if the next line contains the line before at any position because of *"$pre"*. If the line does not contain the line before it will print the line and remember that one as line before.
Change FILENAME with the file name you need.:

#!/bin/sh
pre=
while IFS='' read -r line || [[ -n "$line" ]]; do
   # echo "Line read from file: $line"
   # if line does not contain line pre
   if ! { [ -n "$pre" ] && [[ "$line" == *"$pre"* ]] ; }; then
      # echo "   <$pre> not found!"
      echo "$line"
      pre=$line
   fi
done < "FILENAME"

Output:

abc man
abc 12345
dec asdasfa sadfasfa
dec 22

score 1 · Answer 2 · answered Aug 22 '17 at 08:42

awk '{ dat[NR]=$0 } END { print dat[1];dita=dat[1];for (i=2;i<=NR;i++) { if ( dat[i] !~ dita ) { dita=dat[i];print dita } } }' filename

Broken down:

{ 
 dat[NR]=$0
}
END { 
      print dat[1]
      dita=dat[1]
      for (i=2;i<=NR;i++) {
                          if ( dat[i] !~ dita ) {
                                                  dita=dat[i]
                                                  print dita
                                                }
                          }
    }

Using awk, read all the data into an array and then print the first element of the array (dat) and set the check variable (dita) to the first element. Then loop from the second element to the last checking all data against dita. If there is no pattern match, print the element and set dita to the element just printed.

This will break if lines contain regex characters – 123 Aug 22 '17 at 09:11 — 123, Aug 22 '17 at 09:11

score 1 · Answer 3 · answered Aug 22 '17 at 09:13

Try this script

#! /bin/bash
unset previousLine
while IFS='' read -r currentLine; do
    if [ -z ${previousLine+x} ] || ! grep -qF "$previousLine" <<< "$currentLine"; then
        echo "$currentLine"
        previousLine="$currentLine"
    fi
done < file

[ -z ${previousLine+x} ] is true when $previousLine is unset. grep checks if the previous line is part of the current line.

Edit: Andre Kampling and me both came up with the same solution, but he was the first one to answer the question.

Is it possible to check if a line is included in the next line in unix and if so, delete the next line

3 Answers3