Stack Overflow
Stack Overflow

Questions tagged [uniq]

uniq is a Unix/POSIX/Linux utility to remove or filter duplicate lines from a sorted file. It is also the name of a method to remove duplicates from an array in Ruby.

uniq is a Unix/POSIX/Linux utility to remove or filter duplicate lines from a sorted file. It is typically applied to the output of sort.

In Ruby , uniq is a method of the Array class to remove duplicates from an array. uniq creates a new array whereas uniq! modifies the array in place.

For questions about unique identifiers, keys, names, etc., see or more specific tags such as , , , , etc.

Documentation

454 questions
241
votes
9 answers

Is there a way to 'uniq' by column?

I have a .csv file like this: stack2@domain.example,2009-11-27 01:05:47.893000000,domain.example,127.0.0.1 overflow@domain2.example,2009-11-27 00:58:29.793000000,domain2.example,255.255.255.0 overflow@domain2.example,2009-11-27…
Eno
  • 2,535
  • 2
  • 15
  • 6
155
votes
8 answers

Remove duplicate lines without sorting

I have a utility script in Python: #!/usr/bin/env python import sys unique_lines = [] duplicate_lines = [] for line in sys.stdin: if line in unique_lines: duplicate_lines.append(line) else: unique_lines.append(line) …
Robottinosino
  • 10,384
  • 17
  • 59
  • 97
131
votes
13 answers

Find unique lines

How can I find the unique lines and remove all duplicates from a file? My input file is 1 1 2 3 5 5 7 7 I would like the result to be: 2 3 sort file | uniq will not do the job. Will show all values 1 time
amprantino
  • 1,597
  • 4
  • 15
  • 15
56
votes
6 answers

Sort & uniq in Linux shell

What is the difference between the following to commands? sort -u FILE sort FILE | uniq
yassin
  • 6,529
  • 7
  • 34
  • 39
53
votes
3 answers

How to print only the unique lines in BASH?

How can I print only those lines that appear exactly once in a file? E.g., given this file: mountain forest mountain eagle The output would be this, because the line mountain appears twice: forest eagle The lines can be sorted, if necessary.
Village
  • 22,513
  • 46
  • 122
  • 163
38
votes
1 answer

How to do natural sort output of "uniq -c" in descending/acsending order? - unix

How to do natural sort on uniq -c output? When the counts are <10, the uniq -c | sort output looks fine: alvas@ubi:~/testdir$ echo -e "aaa\nbbb\naa\ncd\nada\naaa\nbbb\naa\nccd\naa" > test.txt alvas@ubi:~/testdir$ cat…
alvas
  • 115,346
  • 109
  • 446
  • 738
22
votes
2 answers

remove white space in bash using sed

I have a file that contains a number followed by a file path on each line for a large amount of files. So it looks like this: 7653 /home/usr123/file123456 But the problem with this is there is 6 empty white spaces before it which throws off…
Sal
  • 625
  • 2
  • 7
  • 11
19
votes
4 answers

sort unique urls from log

I need to get the unique URLs from a web log and then sort them. I was thinking of using grep, uniq, sort command and output this to another file I executed this command: cat access.log | awk '{print $7}' > url.txt then only get the unique one and…
aki
  • 1,241
  • 2
  • 13
  • 43
18
votes
3 answers

Using linux command "sort -f | uniq -i" together for ignoring case

I am trying to find unique and duplicate data in a list of data with two columns. I really just want to compare the data in column 1. The data might look like this (separated by a tab): What are you doing? Che cosa stai facendo? WHAT ARE YOU…
Steve3p0
  • 2,332
  • 5
  • 22
  • 30
16
votes
4 answers

Rails 3, ActiveRecord, PostgreSQL - ".uniq" command doesn't work?

I have following query: Article.joins(:themes => [:users]).where(["articles.user_id != ?", current_user.id]).order("Random()").limit(15).uniq and gives me the error PG::Error: ERROR: for SELECT DISTINCT, ORDER BY expressions must appear in select…
user984621
  • 46,344
  • 73
  • 224
  • 412
16
votes
2 answers

How to sort,uniq and display line that appear more than X times

I have a file like this: 80.13.178.2 80.13.178.2 80.13.178.2 80.13.178.2 80.13.178.1 80.13.178.3 80.13.178.3 80.13.178.3 80.13.178.4 80.13.178.4 80.13.178.7 I need to display unique entries for repeated line (similar to uniq -d) but only entries…
Andrew Kennen
  • 163
  • 1
  • 1
  • 6
14
votes
3 answers

Using Hadoop, are my reducers guaranteed to get all the records with the same key?

I'm running a Hadoop job using Hive actually that is supposed to uniq lines in many text files. In the reduce step, it chooses the most recently timestamped record for each key. Does Hadoop guarantee that every record with the same key, output by…
samg
  • 3,496
  • 1
  • 25
  • 26
13
votes
5 answers

Why does uniq! return nil if there are no duplicates

I'm just starting with Ruby and I personally find the following to be a violation of the "principle of least surprise". And that is, quoting from the documentation, that uniq! "removes duplicate elements from self. Returns nil if no changes are…
Dexygen
  • 12,287
  • 13
  • 80
  • 147
11
votes
5 answers

Changing delimiter of the uniq command

I'd like the output of the uniq command to be comma separated, so that instead of: 30 hello 31 world 36 hey_there 142 i_am_bigest I'll get: 30,hello 31,world 36,hey_there 142,i_am_biggest My input has no spaces, but just using…
eran
  • 14,496
  • 34
  • 98
  • 144
10
votes
6 answers

How to select unique elements

I would like to extend the Array class with a uniq_elements method which returns those elements with multiplicity of one. I also would like to use closures to my new method as with uniq. For example: t=[1,2,2,3,4,4,5,6,7,7,8,9,9,9] t.uniq_elements #…
Konstantin
  • 2,983
  • 3
  • 33
  • 55
1
2 3
…
30 31