Query to detect duplicate rows

Question

I had read data from an XML file into DataSet by using c# than I want to identify duplicate (completely the same ) rows in that set. I tried such kind of grouping and that works!

var d= from r1 in table.AsEnumerable()
       group r1 by new
       {
            t0 = r1[0],
            t1 = r1[1],
            t2 = r1[2],
            t3 = r1[3],
            t4 = r1[4],
            t5 = r1[5],
            t6 = r1[6],
            t7 = r1[7],
            t8 = r1[8],
       }
       into grp
       where grp.Count() > 1
       select grp;

But the number of data columns can be differ, so I cannot apply static grouping in query like above. I had to generate the grouping array dynamically?

I don't want to delete dublicate, I just want to find them!

What you are looking for is a way to compare all properties of an instance. This may help: http://stackoverflow.com/questions/506096/comparing-object-properties-in-c-sharp — TimothyP, Jan 28 '13 at 06:53

abatishchev · Accepted Answer · 2013-01-28T18:51:04.103

6

var rows = table.AsEnumerable();
var unique = rows.Distinct(DataRowComparer.Default);
var duplicates = rows.Except(unique); // , DataRowComparer.Default);

edited Jan 28 '13 at 18:51

answered Jan 28 '13 at 06:53

abatishchev

98,240
88
296
433

@CuongLe: Will this not filter out duplicate rows using `EqualityComparer.Default` which compared all row's columns? – abatishchev Jan 28 '13 at 06:59
No, DataRow is not like anonymous class, not structural type – cuongle Jan 28 '13 at 07:01
@CuongLe: okay, use `DataRowComparer.Default` then. – abatishchev Jan 28 '13 at 07:14
1

The `duplicates` collection is always empty because the same comparer that is used to detect duplicates, removes all the occurances of that duplicate row (no only the first one). – Mohammad Dehghan Jan 28 '13 at 07:19

score -1 · Answer 2 · answered Jan 28 '13 at 06:56

Try this

DataTable dt = TableName;
dt = oPerformance.RemoveDuplicateRows(dt, ColToCompareAndRemove);

and here is the function RemoveDuplicateRows

public DataTable RemoveDuplicateRows(DataTable dTable, string colName)
        {
            Hashtable hTable = new Hashtable();
            ArrayList duplicateList = new ArrayList();

            //Add list of all the unique item value to hashtable, which stores combination of key, value pair.
            //And add duplicate item value in arraylist.
            foreach (DataRow drow in dTable.Rows)
            {
                if (hTable.Contains(drow[colName]))
                    duplicateList.Add(drow);
                else
                    hTable.Add(drow[colName], string.Empty);
            }

            //Removing a list of duplicate items from datatable.
            foreach (DataRow dRow in duplicateList)
                dTable.Rows.Remove(dRow);

            //Datatable which contains unique records will be return as output.
            return dTable;
        }

Use `List<>` and `Dictionary<,>` instead. Are you aware about generics' benefits? — abatishchev, Jan 28 '13 at 06:58

Query to detect duplicate rows

2 Answers2