Printing duplicate rows as many times it is duplicate in the input file using UNIX

Question

Suppose I have a sorted file:

 AARAV,12345,BANK OF AMERICA,$145
 AARAV,12345,BANK OF AMERICA,$145
 AARAV,12345,BANK OF AMERICA,$145
 RAM,124455,DUETCHE BANK,$240

And I want output as:

 AARAV,12345,BANK OF AMERICA,$145
 AARAV,12345,BANK OF AMERICA,$145

With **uniq -d file** I am able to find duplicate records but its printing the record only once even if it is repeated. I want to print as many times it is duplicated. Thanks in advance.

Can anyone advise me here.. if input line has a star(*) or any regex character grep will not work. What else can I do here. — Aarav, Apr 18 '14 at 07:01
Try using `grep -F` isntead. See here for details: http://stackoverflow.com/questions/3242873/grep-for-literal-strings — merlin2011, Apr 18 '14 at 07:05

merlin2011 · Accepted Answer · 2014-04-12T08:58:00.927

1

The following should do what you want, assuming your file is called Input.txt.

uniq -d Input.txt  | xargs -I {} grep   {} Input.txt

xargs -I {} basically tells xargs to substitute the input that is being piped in whenever it sees {} in a later command.

grep {} Input.txt will be called with each line of input from the pipe, where the line of input will get substituted where {} is.

Why does this work? We are using uniq -d to find the duplicate entries, and then using them as input patterns to grep to match all the lines which contain those entries. Thus, only duplicate entries are printed, and they are printed exactly as many times as they appear in the file.

Update: Printing the duplicates occurences only, not the first occurrence, in a way that is compatible with ksh, since the OP does not apparently have bash on his system.

uniq -d Input.txt | xargs -L 1 | while read line
do
    grep  "$line"  Input.txt | tail -n +2; 
done

Note that in the above scripts, we are assuming that no line is a substring of another line.

edited Apr 12 '14 at 08:58

answered Apr 12 '14 at 04:30

merlin2011

71,677
44
195
329

Hey can you please explain your command. What are these braces meant to do – Aarav Apr 12 '14 at 04:54
Thanks for the detailed info. With uniq we will find which rows are duplicate, and with grep we will search them in input file. Thats good but my requirement is not to print the first time it appears. Like 123 is present 4 times in input file. so, thrice it is duplicate. So it should come thrice not 4 times. – Aarav Apr 12 '14 at 05:09
@Aarav, Updating again. – merlin2011 Apr 12 '14 at 05:20
I m sorry.. I am writing this in unix shell script. But its giving me error..Am I interpreting this wrong. – Aarav Apr 12 '14 at 06:46
@Aarav, I tested this the `bash` shell. What shell are you using? – merlin2011 Apr 12 '14 at 07:27
I am using ksh shell. – Aarav Apr 12 '14 at 07:49
@Aarav, Try typing `bash` on your command line, pushing `Enter` and then using the commands in my answer. – merlin2011 Apr 12 '14 at 07:51
I dnt have bash in my system :-( – Aarav Apr 12 '14 at 07:59
What flavor of *nix are you running? This is the first time I've heard of a nix without bash? – merlin2011 Apr 12 '14 at 08:00
If you aren't sure, then `cat /etc/issue` *might* tell you. – merlin2011 Apr 12 '14 at 08:01
@Aarav, I updated again with a solution that I tested in `ksh`. Throw it into a script and see if it also runs on your system. – merlin2011 Apr 12 '14 at 08:19
This will work good with the above file but for a file having some 130 characters with a space in them. it will read it as new line when we encounter a space – Aarav Apr 12 '14 at 08:46
You can't do multiline comments, you will have to update your question. – merlin2011 Apr 12 '14 at 08:47
Updated the question with the actual data to process. – Aarav Apr 12 '14 at 08:52

score 0 · Answer 2 · answered Apr 12 '14 at 04:46

0

This should give you the output that you want. It repeats each duplicate line N-1 times. Unfortunately the output isn't sorted, so you'd have to pipe it through sort again.

Assuming the input file is input.txt:

awk -F '\n' '{ a[$1]++ } END { for (b in a) { while(--a[b]) { print b } } }' input.txt | sort

answered Apr 12 '14 at 04:46

Bradley Snyder

228
4
7

This is so great. But the example I have given is different. I have a big file having some 130 characters in a line with some space as well. This is not working for that. – Aarav Apr 12 '14 at 05:00
Can you share an example of some data that doesn't work with it? – Bradley Snyder Apr 12 '14 at 05:26
Example: AARAVA,ght,BANK OF AMERICA,123 – Aarav Apr 12 '14 at 08:45

Printing duplicate rows as many times it is duplicate in the input file using UNIX

2 Answers2