Remove List elements that appear more than once, in place

Question

There is a similar question posted, but I do not have the rep to ask a follow-up question in that thread. :(

If I have a List<T> that contains items that appear more than once, List.Distinct() will remove duplicates, but the original will still remain in place. If I want to remove items that occur more than once, including the original, what would be the most efficient way to do this to the original list?

Given a List<int> called oneTime:

{ 4, 5, 7, 3, 5, 4, 2, 4 }

The desired output would be in oneTime:

{ 7, 3, 2 }

---

Follow up question for @Enigmativity:

Here is a pseudo version of what my script is doing. It is done in NinjaTrader which runs on .NET3.5.

I will attach a general idea of what the code is supposed to be doing, I'd attach the actual script but unless using NinjaTrader, it might not be of use.

But essentially, there is a large z loop. Each time through, a series of numbers is added to 'LiTics.' Which I do not want to disturb. I then pass that list to the function, and return a list of values that only occur once. Then I'd like to see those numbers each time through the loop.

It works initially, but running this on various sets of data, after a few passes through the loop, it start reporting values that occur more than once. I'm not sure why exactly?

for(int z=1; z<=10000; z +=1)//Runs many times 
{ 
    if (BarsInProgress ==0 &&CurrentBar-oBarTF1>0 &&startScript )   //Some Condition
    {
        for(double k=Low[0]; k<=High[0]; k +=TickSize)  
        {   
            LiTics.Add(k);  
            //Adds a series of numbers to this list each time through z loop
            //This is original that I do not want to disturb
        }

        LiTZ.Clear();  //Display list to show me results Clear before populating
        LiTZ=GetTZone(LiTics); //function created in thread(below)
                               //Passing the undisturbed list that is modified on every loop
        foreach (double prime in LiTZ) { Print(Times[0] +",  " +prime);  }
        //Printing to see results   
    }

}//End of bigger 'z' loop

//Function created to get values that appear ONLY once
public List<double> GetTZone(List<double> sequence) 
{  
    var result =
        sequence
            .GroupBy(x => x)
            .Where(x => !x.Skip(1).Any())
            .Select(x => x.Key)
            .ToList();
    return result;
}

A picture of the print out and what is going wrong: Screenshot.

Answer 1

So, if you can have a new list, then this is the easiest way to do it:

var source = new List<int>() { 4, 5, 7, 3, 5, 4, 2, 4 };

var result =
    source
        .GroupBy(x => x)
        .Where(x => !x.Skip(1).Any())
        .Select(x => x.Key)
        .ToList();

This gives:

{ 7, 3, 2 }

If you want to remove the values from the original source, then do this:

var duplicates =
    new HashSet<int>(
        source
            .GroupBy(x => x)
            .Where(x => x.Skip(1).Any())
            .Select(x => x.Key));

source.RemoveAll(n => duplicates.Contains(n));

Answer 2

Here is an extension method for the List<T> class, that removes from the list all the items that appear more than once:

/// <summary>
/// Removes all the elements that have a key that appears more than once,
/// according to a specified key selector function.
/// </summary>
public static int RemoveDuplicatesByKey<TSource, TKey>(this List<TSource> list,
    Func<TSource, TKey> keySelector,
    IEqualityComparer<TKey> comparer = default)
{
    ArgumentNullException.ThrowIfNull(list);
    ArgumentNullException.ThrowIfNull(keySelector);
    Dictionary<TKey, int> occurences = new(list.Count, comparer);
    foreach (TSource item in list)
        CollectionsMarshal.GetValueRefOrAddDefault(
            occurences, keySelector(item), out _)++;
    return list.RemoveAll(item => occurences.TryGetValue(
        keySelector(item), out int value) && value > 1);
}

The occurrences of each element are counted with a Dictionary<TKey, int>, using the CollectionsMarshal.GetValueRefOrAddDefault method (.NET 6) for efficiency.

Usage example:

List<int> list = new() {4, 5, 7, 3, 5, 4, 2, 4};
Console.WriteLine($"Before: [{String.Join(", ", list)}]");
int removedCount = list.RemoveDuplicatesByKey(x => x);
Console.WriteLine($"After: [{String.Join(", ", list)}], Removed: {removedCount}");

Output:

Before: [4, 5, 7, 3, 5, 4, 2, 4]
After: [7, 3, 2], Removed: 5

Online Demo.

Note: The keySelector delegate should not throw exceptions, otherwise the RemoveDuplicatesByKey might introduce new duplicates instead of removing the existing ones.

Answer 3

I have two options for you, one that uses HashSet and other Linq .

Option 1:

Using HashSet, loop through collection and insert if it not exist and remove if it exists.

HashSet<int> hash = new HashSet<int>();

foreach(var number in list)
{
    if(!hash.Contains(number)) hash.Add(number);
    else hash.Remove(number);               
}
list = hash.ToList();

Option 2:

Simple Linq, group the elements and filter whose count >1.

var list= list.GroupBy(g=>g)
    .Where(e=>e.Count()==1)
    .Select(g=>g.Key)
    .ToList();

There is big performance gain using HashSet over Linq, it is obvious, Linq(in this case) require multiple iterations, where as HashSet uses single iteration and provides LookUp (for adding/removing) with O(1) access.

Elapsed Time (Using Linq): 8808 Ticks
Elapsed Time (Using HashSet): 51 Ticks

Working Demo