Using LINQ to remove duplicate content in list?

Question

Suppose I have this list:

int[] list = { 1, 2, 3, 3, 1 };

What I would like is to remove duplicates that immediately follow same number. So in this case I want to remove 3, but not 1.

New list should therefore be: {1, 2, 3, 1}

Another example is this list: {2, 7, 7, 7, 2, 6, 4} which will become {2, 7, 2, 6, 4}.

Can I do this with LINQ?

score 5 · Accepted Answer · answered Jun 24 '17 at 19:26

You could use Aggregate if you want to use an existing LINQ method but such an approach would lose laziness. You can write your own extension method:

public static IEnumerable<T> RemoveConsecutiveDuplicates<T>(this IEnumerable<T> source, IEqualityComparer<T> comp = null)
{
    comp = comp ?? EqualityComparer<T>.Default;

    using (var e = source.GetEnumerator())
    {
        if (e.MoveNext())
        {
            T last = e.Current;
            yield return e.Current;

            while (e.MoveNext())
            {
                if (!comp.Equals(e.Current, last))
                {
                    yield return e.Current;
                    last = e.Current;
                }
            }
        }
    }
}

I think this is the best LINQ-ish solution because it iterates the source only once. — Gert Arnold, Jun 24 '17 at 19:40

score 3 · Answer 2 · answered Jun 24 '17 at 19:29

If you insist on doing this with LINQ, you could use Aggregate:

var result = array.Aggregate(new List<int>(), (a, b) =>
{
    if (!a.Any() || a.Last() != b)
        a.Add(b);
    return a;
});

But this isn't necessarily the most efficient solution because of Any and Last in each iteration. A simple foreach comparing the previous and current iteration value will perform much better.

Yacoub Massad · Answer 3 · 2017-06-24T19:32:40.137

You can use PairWise from MoreLINQ like this:

var result =
    new[] {list[0]}
        .Concat(
            list
                .Pairwise((x, y) => new {Item = y, Same = x == y})
                .Where(x => !x.Same)
                .Select(x => x.Item))
        .ToArray();

PairWise allows you to get a sequence that results from applying a function on each item in the original sequence along with the item before it (expect for the first item).

What I am doing here is for each item (expect the first item), I am getting the item itself and a boolean (Same) indicating whether this item equals the item before it. Then, I am filtering the sequence to take only the items that each does not equal the item before it. I am then simply appending the first item in the original list to the new sequence.

Note: don't forget to handle the case where list is empty.

NtFreX · Answer 4 · 2017-06-24T20:35:07.747

You could do the following (without linq).

var collection = new [] { 2, 7, 7, 7, 2, 6, 4 }.ToList();
for (int i = 0; i < collection.Count - 1; i++)
{
    if (collection[i] == collection[i + 1])
    {
        collection.RemoveAt(i);
        i--;
    }
}

Another yield based solution would be this.

public static IEnumerable<T> RemoveConsecutiveDuplicates<T>(this IEnumerable<T> collection)
{
    using (var enumerator = collection.GetEnumerator())
    {
        bool wasNotLast = enumerator.MoveNext(), 
             hasEntry = wasNotLast;
        T last = hasEntry ? enumerator.Current : default(T);

        while(wasNotLast)
        {
            if (!last.Equals(enumerator.Current))
                yield return last;

            last = enumerator.Current;
            wasNotLast = enumerator.MoveNext();
        }

        if (hasEntry)
            yield return last;
    }
}

I think you mistyped in line 4, where "array" should be "collection". — brinch, Jun 24 '17 at 20:22

Using LINQ to remove duplicate content in list?

4 Answers4