How to remove duplicate rows from nested list?

Question

I have nested list to perform as 2D array:

public List<List<float?>> arrayFloatValue = new List<List<float?>>(); 

This nested list has 2000 columns in parent and 20000 float values in child array. Now I want to match repeated rows and remove from sub list. Below is sample code.

//Define capacity
int ColumnsCount = 2000;
int RowsCount = 20000;

//Create object
public List<List<float?>> arrayFloatValue = new List<List<float?>>();

//Initiate parent list i.e. 2000 
arrayFloatValue = new List<float?>[ColumnsCount].ToList();

//Initiate child list i.e. 20000 
for (int i = 0; i < ColumnsCount; i++)
{
    arrayFloatValue[i] = new float?[RowsCount].ToList();
}

//Fill dummy data.
for (int x = 0; x < ColumnsCount; x++)
{
    for (int y = 0; y < RowsCount; y++)
    {
        if (y % 50 != 0)
            arrayFloatValue[x][y] = x + y; // Assign dummy value
        else
            arrayFloatValue[x][y] = 0;     // Forcefully 0 value added for each 50th row.
    }
}

Now I have array like

//  [0] [0]     [1] [0]     [2] [0]     ...
//      [1]         [2]         [3]     ...
//      [2]         [3]         [4]     ...
//      [3]         [4]         [5]     ...
//      [4]         [5]         [6]     ...
//      [5]         [6]         [7]     ...
//      [6]         [7]         [8]     ...
//      [7]         [8]         [9]     ...
//      [8]         [9]         [10]    ...
//      [9]         [10]        [11]    ...
//  ...         ...         ...
//      [49]        [50]        [51]    ...
//      [0]         [0]         [0] ...
//  
//  And so on..
//  

Now I want to remove repeated values in each column. Here in above example I have 0 value as repeated at each row index like 50th, 100th 150th .... so on. I want to remove these rows.

Answer 1

You can try good old Distinct with a custom IEqualityComparer<T> (we are going to compare lists with SequenceEqual):

public class ListComparer<T> : IEqualityComparer<IEnumerable<T>> {
  public bool Equals(IEnumerable<T> x, IEnumerable<T> y) {
    return Enumerable.SequenceEqual(x, y);
  }

  public int GetHashCode(IEnumerable<T> obj) {
    return obj == null ? -1 : obj.Count();
  }
}

Now Distinct:

  List<List<float?>> list = new List<List<float?>>() {
    new List<float?>() { 1, 2, 3},
    new List<float?>() { 4, 5, 6, 7},
    new List<float?>() { 1, 2, 3},
    new List<float?>() { null },
    new List<float?>() { 1, 2, null },
    new List<float?>() { null },
    new List<float?>() { 1, 2 },
  };

  var result = list
    .Distinct(new ListComparer<float?>());

  string report = string.Join(Environment.NewLine,
    result.Select(line => $"{string.Join(", ", line)}"));

  Console.Write(report);

Outcome:

1, 2, 3
4, 5, 6, 7

1, 2, 
1, 2

Answer 2

If I understand your question, you can use a HashSet to filter your list. But you have to define a IEqualityComparer> that checks the equality of the elements.

I did an example:

using System;
using System.Collections.Generic;
using System.Linq;

namespace MyNamespace
{
    public class Program
    {
        public static void Main()
        {
            List<List<float>> arrayFloatValue = new List<List<float>>
            {
                new List<float> {1, 2, 3},
                new List<float> {1, 3, 2},
                new List<float> {1, 2, 3},
                new List<float> {3, 5, 7}
            };

            var hsArrayFloatValue = new HashSet<List<float>>(arrayFloatValue, new ListComparer());
            List<List<float>> filteredArrayFloatValue = hsArrayFloatValue.ToList();

            DisplayNestedList(filteredArrayFloatValue);

            //output:
            //1 2 3
            //1 3 2
            //3 5 7
        }

        public static void DisplayNestedList(List<List<float>> nestedList)
        {
            foreach (List<float> list in nestedList)
            {
                foreach (float f in list)
                    Console.Write(f + " ");

                Console.WriteLine();
            }

            Console.ReadLine();
        }
    }

    public class ListComparer : IEqualityComparer<List<float>>
    {
        public bool Equals(List<float> x, List<float> y)
        {
            if (x == null && y == null)
                return true;

            if (x == null || y == null || x.Count != y.Count)
                return false;

            return !x.Where((t, i) => t != y[i]).Any();
        }

        public int GetHashCode(List<float> obj)
        {
            int result = 0;

            foreach (float f in obj)
                result |= f.GetHashCode();

            return result;
        }
    }
}

Although, I don't recommend you to compare floats. Use decimals instead.