4

I have a data file for fortune that contains many repeated fortunes. I would like to remove them.

Fortunes are delineated by %'s, so a sample fortune file may look like this:

%
This is sample fortune 1
%
This is 
sample fortune 2
%
This fortune 
is repeated
%
This is sample fortune 3
%
This fortune 
is repeated
%
This fortune
is unique
%

As you can see, fortunes can span across multiple lines, rendering the solutions here useless.

What can I do to find and remove the repeated fortunes? I thought about just finding a way to make awk ignore lines beginning with %, but some fortunes share identical lines but are not the same overall (such as the last two in my example), so that is not enough.

I've been trying to solve this with awk so far, but any tool is fine.

Community
  • 1
  • 1
SnoringFrog
  • 1,479
  • 1
  • 16
  • 30

2 Answers2

4

That's a job for awk:

awk 'seen[$0]{next}{seen[$0]=1}1' RS='%' ORS='%' fortune

RS='%' means we are using % as the record separator.

seen[$0] checks if we already have seen this value. $0 is the whole record, the fortune's text, as string. If we've seen the value we are moving to the next record and don't print anything.

{seen[$0]=1} adds the record to the lookup table. 1 prints the current record since it is always true. Note that this code gets only executed when we've not seen the record before, because of the next statement before.

ORS='%' set's the output record separator to %.

hek2mgl
  • 152,036
  • 28
  • 249
  • 266
4

Awk can handle it. Set the record separator to "%\n" and then print unique entries:

awk 'BEGIN{RS="%\n"} { if (! ($0 in fortunes)) { fortunes[$0]++; print $0 "%"} }' data
%
This is sample fortune 1
%
This is 
sample fortune 2
%
This fortune 
is repeated
%
This is sample fortune 3
%
This fortune
is unique
%
$
Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278