How to remove duplicated line from a file

Asked May 22 '14 at 13:17

Active May 22 '14 at 13:17

Viewed 72 times

I have a file containing data.

This file could contains duplicated lines.

The line could be repeated many times in the files

I want to remove the duplicated lines (Of curse I have to keep only one copy of a line if the line is duplicated)

For example if my file is containing:

aaa1,bbb1,ccc1
aaa2,bbb2,ccc2
aaa1,bbb1,ccc1
aaa3,bbb3,ccc3
aaa2,bbb2,ccc2

After executing the command the file should contain:

aaa1,bbb1,ccc1
aaa2,bbb2,ccc2
aaa3,bbb3,ccc3

How to do that with linux shell bash?

asked May 22 '14 at 13:17

MOHAMED

1

Ask your professor, or "man sort". You could use associative arrays, too, but I'm guessing that's beyond where you are in class. – Bruce K May 22 '14 at 13:48
Look up sort and uniq – May 22 '14 at 14:07
1

Unleash the power of `awk` ... `awk -F, '!a[$1]++' file` – jaypal singh May 22 '14 at 14:15
Try: `sort yourfile | uniq` – sat May 22 '14 at 14:16

0 Answers0