sed comand to remove duplicate Records from file witout sorting

Question

source file

A B C
E F G
A B C

o/p Should be

A B C
E F G

i dont want to use sort comand as it is adding blank line in the end.I used this

sed '$!N; /^\(.*\)\n\1$/!P; D'

but it required sorted data any alternative to sort the data instead of sed command..?

I guess this question has been answered here >10 times. – Kent Aug 29 '16 at 12:06 — Kent, Aug 29 '16 at 12:06

score 1 · Answer 1 · answered Aug 29 '16 at 12:04

1

awk is better suited for this by using an associative array with key as record itself:

awk '!seen[$0]++' file

A B C
E F G

answered Aug 29 '16 at 12:04

anubhava

I Tried awk also but it is adding blank line in the end I am giving actual scenario $ cat FoundationIssuerPlusDailyChange_ObjectRelType.txt_bkp 1'~'Issuer to Company'~'137'~'2#@#@#2'~'Contract to Underlying Security'~'21'~'21#@#@#wdm@wuvra96a0540:/opt/informatica/Informatica_Projects/wdm/infa_shared/SrcFiles $ awk '!seen[$0]++' FoundationIssuerPlusDailyChange_ObjectRelType.txt_bkp 1'~'Issuer to Company'~'137'~'2#@#@#2'~'Contract to Underlying Security'~'21'~'21#@#@# wdm@wuvra96a0540:/opt/informatica/Informatica_Projects/wdm/infa_shared/SrcFiles – RAZ Aug 29 '16 at 12:10
1

That is only possible if input file has blank lines. In that case try: `awk 'NF && !seen[$0]++' file` – anubhava Aug 29 '16 at 12:12

1 Answers1