Remove line if field is duplicate

Question

Looking for an awk (or sed) one-liner to remove lines from the output if the first field is a duplicate.

An example for removing duplicate lines I've seen is:

awk 'a !~ $0; {a=$0}'

Tried using it for a basis with no luck (I thought changing the $0's to $1's would do the trick, but didn't seem to work).

You asked to remove lines 'if the first field matches' ... what? I've assumed 'the same value as the first field in some previous input line'; another person assumed 'some particular pattern'. What did you intend? — Jonathan Leffler, Apr 08 '10 at 23:24
Your changed version `awk 'a !~ $1; {a=$1}'` *works for me* for adjacent duplicates (e.g. a sorted file). **Jonathan Leffler's** version has the advantage that it will work to remove duplicates on an unsorted file, but at the expense of creating a potentially large array. — Dennis Williamson, Apr 08 '10 at 23:43
I think my main problem was that I was dealing with a few different types of field seperators and wasn't defining FS properly — Kyle, Apr 09 '10 at 15:36

score 26 · Accepted Answer · answered Apr 08 '10 at 23:19

26

awk '{ if (a[$1]++ == 0) print $0; }' "$@"

This is a standard (very simple) use for associative arrays.

answered Apr 08 '10 at 23:19

Jonathan Leffler

That worked! I had another bug that I didn't realize as well that may have been giving me problems aswell. Thanks! – Kyle Apr 08 '10 at 23:25

score 11 · Answer 2 · answered Apr 08 '10 at 23:51

11

this is how to remove duplicates

awk '!_[$1]++' file

answered Apr 08 '10 at 23:51

ghostdog74

3

Using '_' as the array name invites misunderstanding - but it works. – Jonathan Leffler Apr 09 '10 at 00:07

Chris Koknat · Answer 3 · 2015-12-14T19:39:52.737

1

If you're open to using Perl:

perl -ane 'print if ! $a{$F[0]}++' file

-a autosplits the line into the @F array, which is indexed starting at 0
The %a hash remembers if the first field has already been seen

This related solution assumes your field separator is a comma, rather than whitespace

perl -F, -ane 'print if ! $a{$F[0]}++' file

edited Dec 14 '15 at 19:39

answered Sep 15 '15 at 17:24

Chris Koknat

score 0 · Answer 4 · answered Jul 06 '19 at 06:15

0

it print the unique as well as single value of the duplicates

awk '!a[$1]++' file_name

answered Jul 06 '19 at 06:15

Ravi Saroch

4 Answers4