Use awk or sed to eliminate repeat entries in text file

Question

I have a text file like this...

apples
berries
berries
cherries

and I want it to look like this...

apples
berries
cherries

That's it. I just want to eliminate doubled entries. I would prefer for this to be an awk or sed "one-liner" but if there's some other common bash tool that I have overlooked that would be fine.

possible duplicate of [Remove duplicate entries using a Bash script](http://stackoverflow.com/questions/9377040/remove-duplicate-entries-using-a-bash-script) — Avinash Raj, Mar 01 '15 at 10:24
I am piping output from /usr/share/dict/words so it is always in alphabetical order and the duplicates are always adjacent. But if I start using a different dictionary in the future they might not be. — Oswald Roswell, Mar 01 '15 at 13:15

score 5 · Accepted Answer · answered Mar 01 '15 at 10:24

5

sort -u file

if in case you are not worried about the order of the output.

Remove duplicates by retaining the order:

awk '!a[$1]++' file

answered Mar 01 '15 at 10:24

Guru

16,456
2
33
46

score 2 · Answer 2 · answered Mar 01 '15 at 11:09

2

There is a special command for this task, called uniq:

$ uniq file
apples
berries
cherries

This requires that common lines are adjacent, not adjacent equal lines are not removed.

answered Mar 01 '15 at 11:09

user000001

32,226
12
81
108

Use awk or sed to eliminate repeat entries in text file

2 Answers2