How to remove duplicates in List of Class with Linq

Question

I have a list of class :

class GroupAssets
    {
        public string Name { get; set; }
        public List<string> Assets { get; set; }
    }

        List<GroupAssets> GroupList2 = new List<GroupAssets>{
                new GroupAssets { Name="Group1", Assets = new List<string>{ "A","B","C","D" }},
                new GroupAssets { Name="Group1", Assets = new List<string>{ "A","B","E","F" }},
                new GroupAssets { Name="Group3", Assets = new List<string>{ "A","B","H","G" }},
                new GroupAssets { Name="Group4", Assets = new List<string>{ "A","I","C","J" }}
    };

I would like to remove the duplicates and have this result :

Group1 => D
Group2 => E,F
Group3 => H,G
Group4 => I,J
Duplicate => A,B,C

Thank you for your help

All that has been posted is a program description, but that doesn't tell us what _problem_ you're having. What have you tried, and what troubles did you encounter? Please [edit] your post to include a [valid question](//stackoverflow.com/help/how-to-ask) that we can answer. Reminder: make sure you know what is [on-topic](//stackoverflow.com/help/on-topic); asking us to write the program for you, suggestions, and external links are off-topic. — gunr2171, Jan 14 '22 at 14:17
so i suppose your list is new List{ "A","B","C","D" }? and you have group2 instead group1? could you fix your errors please — Frenchy, Jan 14 '22 at 14:28

score 0 · Answer 1 · edited Jan 14 '22 at 16:42

Assuming that you have class GroupAssets { public String Name { get; set; } public IList Assets { get; set; } }

List<GroupAssets> GroupList = new List<GroupAssets>{
  new GroupAssets { Name="Group1", Assets = new List<string>{ "A" ,"B", "C", "D" }},
  new GroupAssets { Name="Group2", Assets = new List<string>{ "A" ,"B", "E", "F" }},
  new GroupAssets { Name="Group3", Assets = new List<string>{ "A" ,"B", "H", "G" }},
  new GroupAssets { Name="Group4", Assets = new List<string>{ "A" ,"I", "C", "J" }},
};

note, that each Asset has 4 items (not 1) you can put

Code:

HashSet<string> duplicates = new HashSet<string>();
HashSet<string> all = new HashSet<string>();

foreach (var item in GroupList)
  foreach (var asset in item.Assets) 
    if (!all.Add(asset))     // duplicate if all contains the asset
      duplicates.Add(asset);

// removing duplicates from each Asset
foreach (var item in GroupList)
  item.Assets.RemoveAll(item => duplicates.Contains(item));

Let's have a look:

string report = string.Join(Environment.NewLine, GroupList
  .Select(item => $"{item.Name} => {string.Join(", ", item.Assets)}"));

Console.WriteLine(report);

Console.WriteLine("Duplicate => {string.Join(", ", duplicates)}");

Outcome:

Group1 => D
Group2 => E, F
Group3 => H, G
Group4 => I, J
Duplicate => A, B, C

If, however, each of Assets contains 1 comma separated item, you should add Split and Join:

HashSet<string> duplicates = new HashSet<string>();
HashSet<string> all = new HashSet<string>();

foreach (var item in GroupList)
  foreach (var asset in item.Assets.SelectMany(list => list.Split(','))) 
    if (!all.Add(asset)) 
      duplicates.Add(asset);

foreach (var item in GroupList) {
  item.Assets = item
    .Assets
    .Select(asset => asset.Split(',').Where(c => !duplicates.Contains(c)))
    .Where(asset => asset.Any())
    .Select(asset => string.Join(",", asset))
    .ToList();
}

score 0 · Answer 2 · answered Jan 14 '22 at 14:37

         List<GroupAssets> GroupList = new List<GroupAssets>{ 
                    new GroupAssets { Name="Group1", Assets = new List<string>{ "A","B","C","D" }},
                    new GroupAssets { Name="Group1", Assets = new List<string>{ "A","B","E","F" }},
                    new GroupAssets { Name="Group3", Assets = new List<string>{ "A","B","H","G" }},
                    new GroupAssets { Name="Group4", Assets = new List<string>{ "A","I","C","J" }}
        };
    var assetList = new Dictionary<string,int>();
    foreach (var g in GroupList.Select(x=> x.Assets)) {
         g.ForEach(x=> {
             if (!assetList.ContainsKey(x)) assetList.Add(x,1);
             else assetList[x]++;
         });
    }

    var nonUnique = assetList.Where(x=> x.Value > 1).Select(x=> x.Key).ToList();
    nonUnique.ForEach(x=> { Console.WriteLine(x); });

alternative solution in case you want to know the total amount of duplicates

Dialecticus · Answer 3 · 2022-01-14T15:02:12.127

I assume you made a mistake and property GroupAssets.Assets contains the list of assets (new List<string>() {"A", "B"}) and not the list of comma separated strings with only one string in the list (new List<string>() {"A,B"}).

First you have to figure out what are the duplicates. You could group items by one of strings "A" to "J" and the value int is the number of occurrences of that key in all lists. We take code from another Stack Overflow question, enhanced with one SelectMany because we want to flatten many lists into one.

    var assetCount = GroupList
        .SelectMany(x => x.Assets)
        .GroupBy(x => x)
        .Select(s => new { Asset = s.Key, Count = s.Count() });

Then we make list of duplicates, and a list of groups with unique assets:

    var duplicates = assetCount.Where(x => x.Count > 1).Select(x => x.Asset).ToList();
    
    var uniqueAssetsGroupList = GroupList
        .Select(x => new GroupAssets() { Name = x.Name, Assets = x.Assets.Except(duplicates).ToList() });
    
    foreach (var group in uniqueAssetsGroupList)
        Console.WriteLine(string.Format("{0} => {1}", group.Name, string.Join(",", group.Assets)));

    Console.WriteLine("Duplicate => {0}", string.Join(",", duplicates));

Hi Dialecticus, It works! Great, thank you! Yes sorry, I made a mistake on the list of assets I corrected my post. Thanks again for your help :-) — Fedo, Jan 14 '22 at 15:48

score 0 · Answer 4 · answered Jan 14 '22 at 17:07

find the unique duplicates and then use except to remove the duplicates from the asset list

  [Fact]
        public void TestRemoveDuplicate()
        {
            List<GroupAssets> GroupList = new List<GroupAssets>{
            new GroupAssets { Name="Group1", Assets = new List<string>{ "A" ,"B", "C", "D" }},
            new GroupAssets { Name="Group2", Assets = new List<string>{ "A" ,"B", "E", "F" }},
            new GroupAssets { Name="Group3", Assets = new List<string>{ "A" ,"B", "H", "G" }},
            new GroupAssets { Name="Group4", Assets = new List<string>{ "A" ,"I", "C", "J" }},
            };
            IList<String> duplicates = new List<String>();
            foreach (var item in GroupList)
            {
                foreach (var element in item.Assets)
                {
                    if (GroupList.Where(e =>e.Name!=item.Name && e.Assets.Contains(element)).Any())
                    {
                        if (duplicates.Contains(element) == false) { duplicates.Add(element); }
                    }
                }
            }
            foreach (var item in GroupList)
            {
                item.Assets = item.Assets.Except(duplicates).ToList();
                string result = "";
                foreach (var element in item.Assets)
                {
                    result += element + " ";
                }
                _output.WriteLine($"Name: {item.Name} Asset: {result}");
            }
            Assert.True(duplicates.Count() > 0);
        }

output:

Name: Group1 Asset: D 
Name: Group2 Asset: E F 
Name: Group3 Asset: H G 
Name: Group4 Asset: I J

score 0 · Answer 5 · answered Jan 27 '22 at 11:46

List<string> tempList =  new List<string>();
Dictionary<string, int> keyValuePairs = new Disctionary<string, int>();

GroupList.ForEach(x => {
    tempList.AddRange(x.Assets);
});

tempList.ForEach(X => {
    if(!keyValuePairs.Keys.Contains(x))
    {
        keyValuePairs.Add(x,1);
    }
    else 
    {
        keyValuePairs[x]++;
    }
});

tempList.Clear();

tempList.AddRange(keyValuePairs.Where(x => x.Value > 1).Select(x => x.Key));

GroupList.ForEach(x => {
    var temp = x.Assets;
    x.Assets = temp.Except(tempList).ToList();
});

How to remove duplicates in List of Class with Linq

5 Answers5