C# Remove duplicates from two lists with multiple properties

Question

I want to insert random elements into a existing database. Before I insert those random elements I have to check if they already exist in the database (primary key). I don't want to add an extra ID attribute.

Therefore I want to get the duplicates but only the duplicate primary keys, therefore .Union() does not work.

Here are two solutions:

var databaseList = database.Repository.GetAllElements();
foreach (var element in listOfRandomElements.Where(element => databaseList.Find(
                x =>
                x.Property0 == element.Property0 &&
                x.Property1 == element.Property1 &&
                x.Property2 == element.Property2)
                != null))
 {
     listOfElements.Remove(element);
 }


 listOfElements.RemoveAll(x => x.Property0 == databaseList.Select(y => true).Property0);

The first solution does not work, because I want to delete elements during the enumeration. I can fix this problem with a workaround (Intelligent way of removing items from a List<T> while enumerating in C#) but for me the whole expression looks horrible.

The second solution (last line of code) does not exist, but something like that would be great.

I hope you have better solutions, I would like to work with LINQ. Perfomance is secundary.

I thought about merging both lists and group them by the primary key, but then I have to delete the elements which have been in the databaseList, but how do I know which elements were in the databaseList.

As always there are hundreds ways of solving this problem, at leat I found one (first solution with a workaround), but maybe there's a more elegant way.

score 3 · Accepted Answer · edited May 23 '17 at 11:58

For those who are looking for a complete solution. Special Thank to @Christos!

var databaseList = database.Repository.GetAllElements();
var keys = databaseList.Select(x => new {x.key0, x.key1, x.key2}); 

listOfElements.RemoveAll(
                x =>
                keys.Any(
                k => k.key0 == x.key0 && 
                k.key1 == x.key1 && 
                k.key2 == x.key2 ));

Found hints here: LINQ to Entities - where..in clause with multiple columns

EDIT: It's not necessary to extract the keys first. It's possible to use the full databaseList instead of keys.

Christos · Answer 2 · 2014-08-20T09:51:23.127

2

Initially you could take a list of all the ids you already have:

var ids = data.Repository.GetAllElements().Select(x=>x.Id);

Then you could simple do the following:

foreach(var element in listOfRandomElements)
{
    // Check if there isn't any id in the ids collection which is equal to element's id.
    if(!ids.Any(x=>x==element.Id)
        // Here you could insert your element.
}

Update

listOfRandomElements.RemoveAll(x=>ids.Contains(x.Id));

edited Aug 20 '14 at 09:51

answered Aug 20 '14 at 09:32

Christos

53,228
8
76
108

Better to use a `HashSet<>` than a list. – phoog Aug 20 '14 at 09:44
thx! that looks like a more readable solution, but I don't want to insert them immediately, only delete them, for modularity reasons. (same problem as in my first solution). – 0lli.rocks Aug 20 '14 at 09:44
@Olli1511 allrigth then. You could do so in the `if` statement, provided that you will change the negation of `ids.Any`. – Christos Aug 20 '14 at 09:46
I think this will cause a InvalidOperationException, because you want to delete a element while enumerating -> same problem as in my solution -> needs a workaround. – 0lli.rocks Aug 20 '14 at 09:50
@Christos Thx! I forgot about the .Contains() method. Now I just need to add more than one property (composite primary key) but I can do this on my own! Thx for help. – 0lli.rocks Aug 20 '14 at 10:02

C# Remove duplicates from two lists with multiple properties

2 Answers2