Find items with duplicate values over mutiple properties

Question

It's hard to explain my issue without giving a concrete example. There might be a similar question on here but I wasn't able to find it because I'm having trouble wording it in searchable terms.

Basically I need to find items in a list that have any duplicate values over multiple properties. In other words, any value in the original list has to be unique regardless of which property it is in.

Here is a simple example I could come up with that describes the problem really well:

There is a list of dates for holidays with an extra property for an optional replacement date (ex: for when the holiday falls in the weekend). Each date in this list has to be unique, so I'd like to find the items that contain duplicate dates.

PART1: return a list of duplicate dates
PART2: return a list of all the items with a duplicate date

I believe this is a great example because one of the properties is nullable which might make it even a little more difficult.

Model:

public class Holiday
{
    public Holiday(DateTime hDate, string descr, DateTime? rDate = null)
    {
        HolidayDate = hDate;
        Description = descr;
        ReplacementDate = rDate;
    }

    public DateTime HolidayDate { get; set; }
    public string Description { get; set; }
    public DateTime? ReplacementDate { get; set; }
}

Here is some example data to get you started (and hopefully clear up any confusion)

var list = new List<Holiday>()
{
    new Holiday(new DateTime(2016,1,1),"NEW YEAR 2016"),
    new Holiday(new DateTime(2016,3,27),"EASTER MONDAY 2016", new DateTime(2016,3,28)),
    new Holiday(new DateTime(2016,12,25),"CHRISTMAS DAY 2016", new DateTime(2016,12,26)),
    new Holiday(new DateTime(2017,1,1),"NEW YEAR 2017", new DateTime(2017,1,2)),
    new Holiday(new DateTime(2017,4,17),"EASTER MONDAY 2017"),
    new Holiday(new DateTime(2017,12,25),"CHRISTMAS DAY 2017"),
    new Holiday(new DateTime(2018,1,1),"NEW YEAR 2018"),
    new Holiday(new DateTime(2018,1,1),"DUPLICATE 1"), //example of a duplicate
    new Holiday(new DateTime(2018,1,2),"DUPLICATE 2", new DateTime(2016,1,1)), //example of a duplicate
    new Holiday(new DateTime(2018,1,3),"DUPLICATE 3", new DateTime(2017,1,2)), //example of a duplicate
    new Holiday(new DateTime(2018,1,4),"DUPLICATE 4", new DateTime(2018,1,4)), //example of a duplicate

};

var result = list; //add your code that finds the items with a duplicate value, preferably a linq query

//PART1: return list of the actual duplicate values
//duplcateDates =  "2016-01-01, 2017-01-02, 2018-01-01, 2018-01-04";

//PART2: return a list with the items that have a duplicate item
var reultString = string.Join(", ", result.Select(q => q.Description));
//resultString = "NEW YEAR 2016, DUPLICATE 2, NEW YEAR 2017, DUPLICATE 3, NEW YEAR 2018, DUPLICATE 1, DUPLICATE 4";

For those of you that are lazy and don't feel like working out this example, as long as you understood my issue and can help me by using your own similar example or point me in the right direction, I'd really appreciate it.

Any solution that doesn't involve writing a specific foreach or for-loop that checks each property individually for duplicates will be accepted as an answer.
I'd like to be able to apply the solution of this problem to different similar situations without having to write an entire block of code iterating trough the possibilities.

This is why I'd like to know if this is possible by linq queries. However if you have a generic method or extension that solves problems like this, please share!

Here is a very good explanation:https://stackoverflow.com/questions/18547354/c-sharp-linq-find-duplicates-in-list#18547390 — FrankM, Dec 14 '17 at 10:33
@FrankM - I already knew how to group on a property and do a count on the possible duplicates. This doesn't really help me much for my problem because there are 2 different properties that has to be checked for. So they can't be equal on the same row, or on other rows. — Oceans, Dec 14 '17 at 10:45
@Evk presumably because a replacement date can't be the same as the holiday date. Tbh seems like there are a few business rules that need addressed here and not just a generic "duplicate" pass. — James, Dec 14 '17 at 10:45
there is no real duplicate in this list when considering all fields(Description is different on each). HolidayDate and ReplacementDate could be used, but its still unclear — FrankM, Dec 14 '17 at 10:45
@TimSchmelter - Yes this is a nullable datetime, just a little added difficulty but not really that important to help me get the actual solution for my question. — Oceans, Dec 14 '17 at 10:46
@Oceans: i have to repeat, is a Replacement-date which is null a duplicate of another Holidays's Replacement-date which is also null? Or should they not be compared then? — Tim Schmelter, Dec 14 '17 at 10:48
@Evk & @TimSchmelter - You guys are correct, the *DUPLICATE 4* could be used solved by defining rules in the model, so maybe there is no need to check for that possibility. As for the uniqueness of each row, just imagine that there is a property for primary key that is just an autoincrement number that defines the uniqueness. Making the `HolidayDate` a unique index or key would solve one of the example cases, but still I'd like to include it in the list. — Oceans, Dec 14 '17 at 10:52
If ignore duplicate 4 (I don't really understand what is it duplicate of) - simple `GroupBy(x => x.ReplacementDate ?? x.HolidayDate)` should do it. — Evk, Dec 14 '17 at 10:52
lol, now i think to understand, you want to match any value of the date. so its a duplicate, too, when ```item1.ReplacementDate == item2.HolidayDate``` — FrankM, Dec 14 '17 at 10:55
@TimSchmelter - `ReplacementDate` is nullable itself but has to be unique only when it has a value ofcourse. *DUPLICATE 3* is a duplicate of *NEW YEAR 2017* because of that property, but with *DUPLICATE 2* it is considered a duplicate of *NEW YEAR 2016*... I hope this explains it? — Oceans, Dec 14 '17 at 10:55
@Evk - the following could be considered another example of a duplicate that can't be discovered with your suggestion `new Holiday(new DateTime(2017,1,1),"DUPLICATE 5", new DateTime(2018,1,5))` this would be a duplicate of *NEW YEAR 2017* — Oceans, Dec 14 '17 at 10:59
Then I misunderstood the requirements. In your last example one holiday really happens on 2018-1-5, but new year 2017 happens on 2017-1-1. — Evk, Dec 14 '17 at 11:02
@FrankM - exactly, basically any date ever mentioned in the list has to be unique. — Oceans, Dec 14 '17 at 11:02
@Evk - I guess when you think logically and look for when *NEW YEAR 2017* takes place you'll first check *2017-01-01* which would lead to *2017-01-02* but this would however give a wrong answer because if you check that date, you'll find *2018-01-05*. Now I want this to be seen as a duplicate so that these can't be entered like this and the correct way to enter it would be like this: `new Holiday(new DateTime(2017,1,1),"NEW YEAR 2017", new DateTime(2018,1,5))` — Oceans, Dec 14 '17 at 11:06
Generally I'd just like to add that I know this is a flawed model and is meant as a fictional situation that describes the problem really well. Just take the model as is, and try to give the desired result. — Oceans, Dec 14 '17 at 11:10
How important is your "Part 1" because I have created a generic method that returns a list of duplicate objects (taking in consideration null values, properties of the same object, and properties of an object in lists). However I didn't make one to return just the duplicated values... — , Dec 14 '17 at 12:54

score 2 · Accepted Answer · answered Dec 14 '17 at 11:18

You can flatten collection so that each date (holiday and replacement) is represeted by seprate item, then group by date, like this:

// flatten
var result = list.SelectMany(c => new[] {
    // always include HolidayDate, it's not nullable
    new {
        Date = c.HolidayDate,
        Instance = c
    },
    // include replacement date if not null
    c.ReplacementDate != null ? new {
        Date = c.ReplacementDate.Value,
        Instance = c
    }: null
})
// filter out null items (that were created for null replacement dates)
.Where(c => c != null)
.GroupBy(c => c.Date)
.Where(c => c.Count() > 1)
.ToArray();

// keys of groups are duplicate dates
var duplcateDates = String.Join(", ", result.Select(c => c.Key.ToString("yyyy-MM-dd")));

var reultString = string.Join(", ", 
      // flatten groups extracting all items
      result.SelectMany(c => c)
      // filter out duplicates
     .Select(c => c.Instance).Distinct()
     .Select(c => c.Description));

I really like this solution a lot. Love the comments that explain everything. Really amazing job. This is a single statement which does exactly what I asked for. Something I can work with to turn into a generic method. — Oceans, Dec 14 '17 at 11:36
Best way. If you want to treat items as equal if one of their properties is equal, the key is that you flatten those properties into a new type which holds each property as a single one and the instance as a second property. Then you can use all LINQ methods (like `GroupBy`) on the single property. I was fiddling around with custom `IEqalityComparer` before i came to exact the same, simple solution. — Tim Schmelter, Dec 14 '17 at 11:49

score 1 · Answer 2 · edited Dec 14 '17 at 11:43

Got one, too: Not sure, if its possible without collecting the data, first.

//PART0: collecting data 
var holidayDateDates = list.Select(x => x.HolidayDate).ToList();
var replacementDates = list.Select(x => x.ReplacementDate).Where(y => y != null).ToList().ConvertAll<DateTime>(x => x.Value);
holidayDateDates.AddRange(replacementDates);
//PART1: return list of the actual duplicate values
var result = holidayDateDates.GroupBy(x => x)
    .Where(g => g.Count() > 1)
    .Select(y => y.Key)
    .ToList();
var duplicateDates = string.Join(", ", result.Select(c => c.ToString("yyyy-MM-dd")));

//PART2: return a list with the items that have a duplicate item
var dummytime = new DateTime();// this will never be in the list and kills all nulls, see below
var duplicateHolidays = list.Where(x => result.Contains(x.HolidayDate) || result.Contains(x.ReplacementDate??dummytime));
var resultString = string.Join(", ", duplicateHolidays.Select(q => q.Description));

You definitely get an A for effort, there is however a small error on this line: `var duplicateDates = string.Join(", ", result.ToString("yyyy-MM-dd"));` this should be replaced with `var duplicateDates = string.Join(", ", result.Select(c => c.ToString("yyyy-MM-dd")));` appart from this small mistake you did a great job, but perhaps a little too many steps to the solution — Oceans, Dec 14 '17 at 11:33

score 1 · Answer 3 · 2017-12-14T16:43:48.843

Here you have a generic extension which returns a list of a custom object that holds a reference to the generic object, the value that is a duplicate, and whether it is a self duplicate or not.

Here is the code that you need to add in a new file:

public class MainObject<TRef, TProp> where TProp : struct
{
    public TRef Reference { get; set; }
    public TProp? Value { get; set; }

    public bool SelfDuplicate { get; set; }

}

public static class Extensions
{
    public static IEnumerable<MainObject<TIn, TProp>> GetDuplicates<TIn, TProp>(this IEnumerable<TIn> @this) where TProp : struct
    {
        var type = typeof(TIn);
        // get the properties of the object type that match the property type
        var props = type.GetProperties(BindingFlags.Instance | BindingFlags.Public).Where(prop => prop.PropertyType == typeof(TProp) || prop.PropertyType == typeof(TProp?)).ToList();

        // convert the input enumerable to a list so we don't iterate it multiple times
        var list = @this as IList<TIn> ?? @this.ToList();

        // create a list to hold all duplicates
        var duplicates = new List<MainObject<TIn, TProp>>();

        foreach (var item1 in list)
        {
            var isSelfDupe = item1.IsDuplicate<TIn, TProp>(props);
            if (isSelfDupe.IsDupe)
            {
                duplicates.Add(new MainObject<TIn, TProp>
                {
                    Reference = item1,
                    SelfDuplicate = true,
                    Value = (TProp?)isSelfDupe.Property1.GetValue(item1)
                });
            }

            foreach (var item2 in list)
            {
                if (ReferenceEquals(item1, item2)) continue;

                var isDuplicate = item1.IsDuplicate<TIn, TProp>(item2, props);
                if (isDuplicate.IsDupe)
                {
                    duplicates.Add(new MainObject<TIn, TProp>
                    {
                        Reference = item1,
                        SelfDuplicate = false,
                        Value = (TProp?)isDuplicate.Property1.GetValue(item1)
                    });
                }
            }
        }

        return duplicates.Distinct().ToList();
    }

    private static IsDuplicateResult<TIn> IsDuplicate<TIn, TProp>(this TIn obj1, IEnumerable<PropertyInfo> props) where TProp : struct
    {
        var propList = props as IList<PropertyInfo> ?? props.ToList();

        var valueList = propList.Select(prop => prop.GetValue(obj1)).ToList();
        foreach (var p1 in propList)
        {
            foreach (var p2 in propList)
            {
                if (ReferenceEquals(p1, p2)) continue;

                if (EqualityComparer<TProp?>.Default.Equals((TProp?)p1.GetValue(obj1), (TProp?)p2.GetValue(obj1)))
                {
                    return new IsDuplicateResult<TIn> { IsDupe = true, Reference = obj1, Property1 = p1, Property2 = p2 };
                }
            }
        }
        return new IsDuplicateResult<TIn> { IsDupe = false };
    }

    private static IsDuplicateResult<TIn> IsDuplicate<TIn, TProp>(this TIn obj1, TIn obj2, IEnumerable<PropertyInfo> props) where TProp : struct
    {
        var propList = props as IList<PropertyInfo> ?? props.ToList();

        var dict1 = propList.ToDictionary(prop => prop, prop => prop.GetValue(obj1));
        var dict2 = propList.ToDictionary(prop => prop, prop => prop.GetValue(obj2));

        foreach (var k1 in dict1.Keys)
        {
            foreach (var k2 in dict2.Keys)
            {
                if (dict1[k1] == null || dict2[k2] == null) continue;

                if (EqualityComparer<TProp?>.Default.Equals((TProp?)dict1[k1], (TProp?)dict2[k2]))
                {
                    return new IsDuplicateResult<TIn> { IsDupe = true, Reference = obj1, Property1 = k1, Property2 = k2 };
                }
            }
        }
        return new IsDuplicateResult<TIn> { IsDupe = false };
    }

    private class IsDuplicateResult<TIn>
    {
        public bool IsDupe { get; set; }

        public TIn Reference { get; set; }

        public PropertyInfo Property1 { get; set; }
        public PropertyInfo Property2 { get; set; }
    }
}

Using this is pretty simple and straightforward, all you have to do is call the extension method.

var result = holidayList.GetDuplicates<Holiday, DateTime>().ToList();
// printing the results
Console.WriteLine(string.Join(", ", result.Select(q => q.Reference.Description)));
// printing the values
var values = result.Select(r => r.Value).Distinct();
Console.WriteLine(string.Join("\r\n", values.Select(q => $"{q:dd/MM/yyyy}")));

DotNetFiddle link: https://dotnetfiddle.net/OGwpb4

DotNetFiddle link (with property annotations): https://dotnetfiddle.net/vzjS2t

You might need some more fiddling around with the functions, to see if anything goes wrong as I didn't have much time to do so but, I think this should get you into the right direction :)

I like the effort but I believe it's not really usable. The main flaw I see in this solution is when the object would contain a third property of the same type that you do not wish to check for. Let's say that the model from the example had another `DateTime` field representing a *CreatedOn* property, you wouldn't want to include this in the search for duplicity. Then your solution can't be used for this.The ideal solution would contain a value selector parameter `params Func[]` — Oceans, Dec 14 '17 at 16:14
@Oceans For that the ideal solution would use annotations. Then all you have to do is add a small check on the props variable to see if they contain an annotation that marks if the fields should be used or not. I think that would be a better looking solution instead of passing multiple expressions. If you want I can implement that for you :) — , Dec 14 '17 at 16:34
@Oceans check this fork: https://dotnetfiddle.net/vzjS2t I've added annotation support to that. Now if you uncomment the annotation at the nullable property, you should get your desired result. I think I'll update the answer to include those changes too later. — , Dec 14 '17 at 16:41

score 0 · Answer 4 · answered Dec 14 '17 at 13:17

In case if anyone, in the future, looking for a generic solution.

Note: This solution will cast nullable properties to their non-nullable types and compare their values too.

Generic solution

For easier implementation we will define two Methods:

IsSelfDuplicate: Which will detect if the object has two properties with the same value.

private bool IsSelfDuplicate(object myObj)
{
    PropertyInfo[] props = myObj.GetType().GetProperties();
    for (int i = 0; i < props.Length; i++)
    {
        if (props[i].GetValue(myObj) == null)
            continue;

        Type iType = (Nullable.GetUnderlyingType(props[i].PropertyType) ?? props[i].PropertyType);
        object iValue = Convert.ChangeType(props[i].GetValue(myObj), iType);

        for (int j = i + 1; j < props.Length; j++)
        {
            if (props[j].GetValue(myObj) == null)
                continue;

            Type jType = (Nullable.GetUnderlyingType(props[j].PropertyType) ?? props[j].PropertyType);
            object jValue = Convert.ChangeType(props[j].GetValue(myObj), jType);

            if (iType == jType && iValue.ToString() == jValue.ToString())
                return true;
        }
    }
    return false;
}

IsDuplicate: Which will detect if two objects, each has a property's value equals in the other.

private bool IsDuplicate(object obj1, object obj2)
{
    PropertyInfo[] props = obj1.GetType().GetProperties();

    for (int i = 0; i < props.Length; i++)
    {
        if (props[i].GetValue(obj1) == null)
            continue;

        Type iType = (Nullable.GetUnderlyingType(props[i].PropertyType) ?? props[i].PropertyType);
        object iValue = Convert.ChangeType(props[i].GetValue(obj1), iType);

        for (int j = 0; j < props.Length; j++)
        {
            if (props[j].GetValue(obj2) == null)
                continue;

            Type jType = (Nullable.GetUnderlyingType(props[j].PropertyType) ?? props[j].PropertyType);
            object jValue = Convert.ChangeType(props[j].GetValue(obj2), jType);

            if (iType == jType && iValue.ToString() == jValue.ToString())
                return true;
        }
    }
    return false;
}

To use the IsSelfDuplicate Method, simply use:

List<Holiday> selfDuplicate = list.Where(l => IsSelfDuplicate(l)).ToList();

To use the IsDuplicate Method, iterate between your list of objects:

List<Holiday> duplicates = new List<Holiday>();

for (int i = 0; i < list.Count; i++)
    for (int j = i + 1; j < list.Count; j++)
    {
         if (IsDuplicate(list[i], list[j]))
         {
              duplicates.Add(list[i]);
              duplicates.Add(list[j]);
              break;
         }
    }

Finally, here is a sample list of objects, as the OP provided it:

var list = new List<Holiday>()
{
     new Holiday(new DateTime(2016,1,1),"NEW YEAR 2016"),
     new Holiday(new DateTime(2016,3,27),"EASTER MONDAY 2016", new DateTime(2016,3,28)),
     new Holiday(new DateTime(2016,12,25),"CHRISTMAS DAY 2016", new DateTime(2016,12,26)),
     new Holiday(new DateTime(2017,1,1),"NEW YEAR 2017", new DateTime(2017,1,2)),
     new Holiday(new DateTime(2017,4,17),"EASTER MONDAY 2017"),
     new Holiday(new DateTime(2017,12,25),"CHRISTMAS DAY 2017"),
     new Holiday(new DateTime(2018,1,1),"NEW YEAR 2018"),
     new Holiday(new DateTime(2018,1,1),"DUPLICATE 1"), //example of a duplicate
     new Holiday(new DateTime(2018,1,2),"DUPLICATE 2", new DateTime(2016,1,1)), //example of a duplicate
     new Holiday(new DateTime(2018,1,3),"DUPLICATE 3", new DateTime(2017,1,2)), //example of a duplicate
     new Holiday(new DateTime(2018,1,4),"DUPLICATE 4", new DateTime(2018,1,4)), //example of a duplicate
};

Now, you will have two results for self and non-self (if I can call it) duplicates. Enjoy ;)

I also like your effort in making this a generic method but I believe that this wouldn't be very useable. As this will compare every property of the same type. The ideal generic method would have some kind of value selectors parameter like: `params Func[]` — Oceans, Dec 14 '17 at 16:09
@Oceans, you can easily add this parameter using the property name. — Bassem, Dec 14 '17 at 20:12

score 0 · Answer 5 · answered Dec 14 '17 at 15:35

You can use IEnumerable and IEnumerator

using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace ConsoleApplication1
{

    class Program
    {
        static void Main(string[] args)
        {
            Holiday holiday = new Holiday();

            foreach (Holiday day in Holiday.holidays)
            {
                foreach (object field in  new HolidayField(day))
                {
                    int a = 1;
                }
                HolidayField hf = new HolidayField(day);
                for (HolidayField.Field field = HolidayField.Field.HOLIDAY_DATE; field < HolidayField.Field.END; field++)
                {
                    object o = hf.GetEnumerator(field);
                }
            }
            var groups = Holiday.holidays.GroupBy(x => x.HolidayDate).ToList();


        }
    }
    public class HolidayField : IEnumerator, IEnumerable
    {
        public Holiday holiday;

        public enum Field
        {
            RESET,
            HOLIDAY_DATE,
            DESCRIPTION,
            REPLACEMENT_DATE,
            END
        }
        Field _holidayProperties;

        public HolidayField(Holiday holiday)
        {
            this.holiday = holiday;

        }

        public Field GetIndex()
        {
            return _holidayProperties;
        }

        public object GetEnumerator(Field field)
        {
            return GetCurrent(holiday, field);
        }

        public HolidayField GetEnumerator()
        {
            return this;
        }

        // Implementation for the GetEnumerator method.
        IEnumerator IEnumerable.GetEnumerator()
        {
            return (IEnumerator) GetEnumerator();
        }



        public object Current
        {
            get
            {
                return GetCurrent(holiday, _holidayProperties);
            }
        }
        public object GetCurrent(Holiday holiday, Field field)
        {
            object results = null;

            switch (field)
            {
                case Field.DESCRIPTION:
                    results = (object)holiday.Description;
                    break;
                case Field.HOLIDAY_DATE:
                    results = (object)holiday.HolidayDate;
                    break;
                case Field.REPLACEMENT_DATE:
                    results = (object)holiday.ReplacementDate;
                    break;
            }

            return results;
        }
        public bool MoveNext()
        {
            _holidayProperties += 1;
            return _holidayProperties == Field.END ? false : true;
        }

        public void Reset()
        {
            _holidayProperties = Field.RESET;
        }
        public void Dispose()
        {
        }

    }
    public class Holiday
    {
        public static List<Holiday> holidays = new List<Holiday>()
        {
            new Holiday(new DateTime(2016,1,1),"NEW YEAR 2016"),
            new Holiday(new DateTime(2016,3,27),"EASTER MONDAY 2016", new DateTime(2016,3,28)),
            new Holiday(new DateTime(2016,12,25),"CHRISTMAS DAY 2016", new DateTime(2016,12,26)),
            new Holiday(new DateTime(2017,1,1),"NEW YEAR 2017", new DateTime(2017,1,2)),
            new Holiday(new DateTime(2017,4,17),"EASTER MONDAY 2017"),
            new Holiday(new DateTime(2017,12,25),"CHRISTMAS DAY 2017"),
            new Holiday(new DateTime(2018,1,1),"NEW YEAR 2018"),
            new Holiday(new DateTime(2018,1,1),"DUPLICATE 1"), //example of a duplicate
            new Holiday(new DateTime(2018,1,2),"DUPLICATE 2", new DateTime(2016,1,1)), //example of a duplicate
            new Holiday(new DateTime(2018,1,3),"DUPLICATE 3", new DateTime(2017,1,2)), //example of a duplicate
            new Holiday(new DateTime(2018,1,4),"DUPLICATE 4", new DateTime(2018,1,4)), //example of a duplicate

        };

        public DateTime HolidayDate { get; set; }
        public string Description { get; set; }
        public DateTime? ReplacementDate { get; set; }


        public Holiday() { }
        public Holiday(DateTime hDate, string descr, DateTime? rDate = null)
        {
            HolidayDate = hDate;
            Description = descr;
            ReplacementDate = rDate;
        }

    }


}

I dreaded that people would suggest solutions like this. I like your intentions but you didn't read the question closely. This might give me the answer to the example from my question, however this wasn't what I was looking for. — Oceans, Dec 14 '17 at 16:12
You title was multiple properties so I assumed you needed a method to enumerate through the properties. You also said " Any solution that doesn't involve writing a specific foreach or for-loop". So how do you get solution without a FOR. Evk solution is doing the same thing I did without the Enumerable and Enumerator. — jdweng, Dec 14 '17 at 16:55

Find items with duplicate values over mutiple properties

5 Answers5