How to get rid of duplicates?

Question

Hi I am writing a script in bash which read the contents of files that have the word "contact"(in the current directory) in them and sorts all the data in those files in alphabetical order and writes them to a file called "out.txt". I was wondering if there was any way in which I could get rid of duplicate content. Any help would be appreciated

The code I have written so far.

#!/bin/bash

cat $(ls | grep contact) > out.txt
sort out.txt -o out.txt

Answer 1

sort has option -u (long option: --unique) to output only unique lines:

sort -u out.txt -o out.txt

EDIT: (Thanks to tripleee)

Your script, at present, contains problems of parsing ls output,

This is a better substitute for what you are trying to do:

sort -u *contact* >out.txt

Answer 2

Use this using the uniq command (easier to remember than flags)

#!/bin/bash

cat $(ls | grep contact) | sort | uniq > out.txt

or the -u flag for sort like this

#!/bin/bash

cat $(ls | grep contact) | sort -u > out.txt

Answer 3

uniq may do what you need. It copies lines from input to output, omitting a line if it was the line it just output.

Answer 4

Take a look at the "uniq" command, and pipe it through there after sorting.