How to sort,uniq and display line that appear more than X times

Question

I have a file like this:

80.13.178.2
80.13.178.2
80.13.178.2
80.13.178.2
80.13.178.1
80.13.178.3
80.13.178.3
80.13.178.3
80.13.178.4
80.13.178.4
80.13.178.7

I need to display unique entries for repeated line (similar to uniq -d) but only entries that occur more than just twice (twice being an example so flexibility to define the lower limit.)

Output for this example should be like this when looking for entries with three or more occurrences:

80.13.178.2
80.13.178.3

score 24 · Answer 1 · answered Nov 22 '13 at 15:01

24

Feed the output from uniq -cd to awk

sort test.file | uniq -cd | awk -v limit=2 '$1 > limit{print $2}'

answered Nov 22 '13 at 15:01

iruvar

22,736
7
53
82

Perfect! Thank you, this one works for me: cat log.txt | sort | uniq -d -c|awk '$1 > 30{print $2}' – Andrew Kennen Nov 22 '13 at 15:04
1

This is a much cleaner, simpler solution.. Thank you – Krishna Vedula Oct 21 '14 at 11:12

score 11 · Accepted Answer · answered Nov 22 '13 at 15:03

11

With pure awk:

awk '{a[$0]++}END{for(i in a){if(a[i] > 2){print i}}}' a.txt

It iterates over the file and counts the occurances of every IP. At the end of the file it outputs every IP which occurs more than 2 times.

answered Nov 22 '13 at 15:03

hek2mgl

152,036
28
249
266

@Kent Thanks to Bell Labs! :) – hek2mgl Nov 22 '13 at 15:09

How to sort,uniq and display line that appear more than X times

2 Answers2

Linked