Extracting string between 2 strings with bash shell script

Question

I've seen questions similar to this, but none of the solutions seem to work in this case. I have a text file that looks something like this

START-OF-FILE
RUNDATE=20140910
FIRMNAME=dl
FILETYPE=pc
REPLYFILENAME=TEST
DERIVED=yes
PROGRAMFLAG=oneshot
SECID=ISIN
SECMASTER=yes
PROGRAMNAME=getdata
START-OF-FIELDS
ISSUER
START-OF-DATA
US345370CN85|0|4|FORD MOTOR COMPANY|FORD MOTOR COMPANY| | |
US31679BAC46|0|4|FIFTH STREET FINANCE COR|FIFTH STREET FINANCE COR| | |
END-OF-DATA
END-OF-FILE

I'm trying to write a bash shell script to extract only the text between "START-OF-DATA" and "END-OF-DATA" excluding both of these. So output i'm looking for would look like this

US345370CN85|0|4|FORD MOTOR COMPANY|FORD MOTOR COMPANY| | |
US31679BAC46|0|4|FIFTH STREET FINANCE COR|FIFTH STREET FINANCE COR| | |

The code i've written so far looks like this

while read line
do
    name=$line

    echo $name | sed -e 's/START-OF-DATA\(.*\)END-OF-DATA/\1/'

done < $1

and running it from bash like

./script.sh file.txt

where script.sh is what I have saved the shell script as and file.txt is the text file above that it reads. At the moment it just reads and echoes the entire file. I'm guessing its something silly in my syntax. Any pointers in the right direction would be much appreciated. Thanks

Answer 1

Using awk you can do:

awk '/START-OF-DATA/{p=1;next} /END-OF-DATA/{p=0;exit} p' file
US345370CN85|0|4|FORD MOTOR COMPANY|FORD MOTOR COMPANY| | |
US31679BAC46|0|4|FIFTH STREET FINANCE COR|FIFTH STREET FINANCE COR| | |

Or using sed:

sed -n '/START-OF-DATA/,/END-OF-DATA/{/START-OF-DATA\|END-OF-DATA/!p;}' file
US345370CN85|0|4|FORD MOTOR COMPANY|FORD MOTOR COMPANY| | |
US31679BAC46|0|4|FIFTH STREET FINANCE COR|FIFTH STREET FINANCE COR| | |

Answer 2

In order to make your solution work you could make a marker when you hit "START-OF-DATA" that reads "True" (or similar), and then end it when you hit "END-OF-DATA". Using this marker you could tell echo to print when the marker reads "True" (when you are inside the relevant block of text).

...or you could use sed:

sed -n '/START-OF-DATA/,/END-OF-DATA/ { //!p }' file.txt

Answer 3

I'd like to add the perlish grep way, as mentioned here:

grep -Pzo "(?s)START-OF-DATA.*END-OF-DATA" "$1"

This still includes the START-OF-DATA and END-OF-DATA markers. To get rid of them, the pattern has to become a bit less readable:

grep -Pzo "(?s)(?<=START-OF-DATA\n).*(?=\nEND-OF-DATA)"

(?<=START-OF-DATA\n) and (?=\nEND-OF-DATA) are look-around assertions as described in perlre, i.e. they are used for matching, but not included in the result.