Extract lines between 2 tokens in a text file using bash

Question

i have a text file which looks like this:

random useless text 
<!-- this is token 1 --> 
para1 
para2 
para3 
<!-- this is token 2 --> 
random useless text again

I want to extract the text in between the tokens (excluding the tokens of course). I tried using ## and %% to extract the data in between but it didn't work. I think it is not meant for manipulating such large text files. Any suggestions how i can do it ? maybe awk or sed ?

Answer 1

No need for head and tail or grep or to read the file multiple times:

sed -n '/<!-- this is token 1 -->/{:a;n;/<!-- this is token 2 -->/b;p;ba}' inputfile

Explanation:

• -n - don't do an implicit print
• /<!-- this is token 1 -->/{ - if the starting marker is found, then
  • :a - label "a"
    • n - read the next line
    • /<!-- this is token 2 -->/q - if it's the ending marker, quit
    • p - otherwise, print the line
  • ba - branch to label "a"
• } end if

Answer 2

You can extract it, including the tokens with sed. Then use head and tail to strip the tokens off.

... | sed -n "/this is token 1/,/this is token 2/p" | head -n-1 | tail -n+2

Answer 3

no need to call mighty sed / awk / perl. You could do it "bash-only":

#!/bin/bash
STARTFLAG="false"
while read LINE; do
    if [ "$STARTFLAG" == "true" ]; then
            if [ "$LINE" == '<!-- this is token 2 -->' ];then
                    exit
            else
                    echo "$LINE"
            fi
    elif [ "$LINE" == '<!-- this is token 1 -->' ]; then
            STARTFLAG="true"
            continue
    fi
done < t.txt

Kind regards

realex

Answer 4

Try the following:

sed -n '/<!-- this is token 1 -->/,/<!-- this is token 2 -->/p' your_input_file
        | egrep -v '<!-- this is token . -->'

Answer 5

Maybe sed and awk have more elegant solutions, but I have a "poor man's" approach with grep, cut, head, and tail.

#!/bin/bash

dataFile="/path/to/some/data.txt"
startToken="token 1"
stopToken="token 2"

startTokenLine=$( grep -n "${startToken}" "${dataFile}" | cut -f 1 -d':' )
stopTokenLine=$( grep -n "${stopToken}" "${dataFile}" | cut -f 1 -d':' )

let stopTokenLine=stopTokenLine-1
let tailLines=stopTokenLine-startTokenLine

head -n ${stopTokenLine} ${dataFile} | tail -n ${tailLines}

Answer 6

For anything like this, I'd reach for Perl, with its combination of (amongst others) sed and awk capabilities. Something like (beware - untested):

my $recording = 0;
my @results = ();
while (<STDIN>) {
   chomp;
   if (/token 1/) {
      $recording = 1;
   }
   else if (/token 2/) {
      $recording = 0;
   }
   else if ($recording) {
      push @results, $_;
   }
}

Answer 7

sed -n "/TOKEN1/,/TOKEN2/p" <YOUR INPUT FILE> | sed -e '/TOKEN1/d' -e '/TOKEN2/d'