How can I take a list of dates, and remove any that are within 1 minute of another date in the list with LINQ?

Question

I'm trying to parse a list of dates so say I have the following data:

2012-02-19 10:06:29.287
2012-02-19 10:06:29.900
2014-01-21 15:21:11.114
2015-04-22 01:11:50.233
2015-04-22 01:11:55.921
2015-04-22 01:12:12.144
2017-12-18 12:01:01.762

I want to then be left with the following list:

2012-02-19 10:06:29.900
2014-01-21 15:21:11.114
2015-04-22 01:12:12.144
2017-12-18 12:01:01.762

Where any dates that are within 1 minute of each other, all are removed except the most recent date. With fluent syntax LINQ if possible.

So in the above example we have 2 dates that fit that criteria:

2012-02-19 10:06:29.287
2012-02-19 10:06:29.900

Are within 1 minute of each other, so the first entry is removed so only the most recent is left.

2015-04-22 01:11:50.233
2015-04-22 01:11:55.921
2015-04-22 01:12:12.144

Has 3 within a minute of each other, so the first two should be removed and left with only the last.

I doubt anyone will write the code for you. You need to try and solve your problem first and then ask for help with what you have tried and what hasn't worked. — Vidmantas Blazevicius, Mar 15 '18 at 19:00
What would happen if you had 100 sequential dates that were 30 seconds apart? — Jonathon Chase, Mar 15 '18 at 19:09
Jonathan Chase: That wouldn't be possible. But I see your point, perhaps the way around that would be first round every date time to the nearest minute and then remove any that occur on the same minute instead. Which is also a lot easier to implement. — SventoryMang, Mar 15 '18 at 19:33
Using my [`TimeSpan.Round`](https://stackoverflow.com/a/41193749/2557128) extension and [`DistinctBy`](https://github.com/morelinq/MoreLINQ/blob/master/MoreLinq/DistinctBy.cs), you can just do `src.DistinctBy(dt => dt.Date+dt.TimeOfDay.Round(TimeSpan.FromMinutes(1)))`. — NetMage, Mar 16 '18 at 18:07

NetMage · Accepted Answer · 2018-03-15T20:28:53.287

Using a few extension methods, you can do this in LINQ, though it isn't ideal.

The first is a variation of the APL scan operator, which is similar to Aggregate, but returns the intermediate results computed from the previous and current values.

public static IEnumerable<KeyValuePair<TKey, T>> ScanPair<T, TKey>(this IEnumerable<T> src, TKey seedKey, Func<KeyValuePair<TKey, T>, T, TKey> combine) {
    using (var srce = src.GetEnumerator()) {
        if (srce.MoveNext()) {
            var prevkv = new KeyValuePair<TKey, T>(seedKey, srce.Current);

            while (srce.MoveNext()) {
                yield return prevkv;
                prevkv = new KeyValuePair<TKey, T>(combine(prevkv, srce.Current), srce.Current);
            }
            yield return prevkv;
        }
    }
}

The second is another APL operator implementation, this time of the compress operator, which uses a boolean vector to select elements.

public static IEnumerable<T> Compress<T>(this IEnumerable<bool> bv, IEnumerable<T> src) {
    using (var srce = src.GetEnumerator()) {
        foreach (var b in bv) {
            srce.MoveNext();
            if (b)
                yield return srce.Current;
        }
    }
}

The third lets you concatenate values to a sequence:

public static IEnumerable<T> Append<T>(this IEnumerable<T> rest, params T[] last) => rest.Concat(last);

Now you can process the list:

var ans = src.Scan((prev, cur) => (cur-prev).TotalSeconds > 60) // find times over a minute apart
             .Append(true) // always keep the last time
             .Compress(src); // keep the DateTimes that are okay

Wow that is some crazy LINQ query there. I came up with a simpler way to do that (in the comments of the question), but this does what the original question asked so you win! Thanks man. — SventoryMang, Mar 15 '18 at 20:06
You can also run into the problem I did with this answer - I forgot I had an extension method that was perfect for making this simpler. I modified my answer to use it. — NetMage, Mar 15 '18 at 20:16
Updated answer again with better extension methods per @JonSkeet with `using`. — NetMage, Mar 15 '18 at 20:29

score 0 · Answer 2 · answered Mar 15 '18 at 19:53

var data = new List<DateTime>
            {
                new DateTime(2012, 02, 19, 10, 06, 29, 287), new DateTime(2012, 02, 19, 10, 06, 29, 900) ,
                new DateTime(2014, 01, 21, 15, 21, 11, 114) ,
                new DateTime(2015, 04, 22, 01, 11, 50, 233),
                new DateTime(2015, 04, 22, 01, 11, 55, 921),
                new DateTime(2015, 04, 22, 01, 12, 12, 144),
                new DateTime(2017, 12, 18, 12, 01, 01, 762)
            };

            var dataFIltered = data.Select(c => new DateTime(c.Year,c.Month,c.Minute)).Distinct().ToList();

This solution removes the seconds and milliseconds from original data — Michael D., Mar 15 '18 at 20:04

score 0 · Answer 3 · answered Mar 15 '18 at 20:04

Here's a small example I wrote up in a console program.. It involves two lists, but gets the job done:

static void Main(string[] args)
    {
        List<DateTime> times = new List<DateTime>();
        times.Add(new DateTime(2012, 02, 19, 10, 06, 29));
        times.Add(new DateTime(2012, 02, 19, 10, 06, 29));
        times.Add(new DateTime(2014, 01, 21, 15, 21, 11));
        times.Add(new DateTime(2015, 04, 22, 01, 11, 50));
        times.Add(new DateTime(2015, 04, 22, 01, 11, 55));
        times.Add(new DateTime(2015, 04, 22, 01, 12, 12));
        times.Add(new DateTime(2017, 12, 18, 12, 01, 01));

        List<DateTime> TheList = new List<DateTime>();

        DateTime min = times.OrderBy(c => c.Date).ThenBy(c => c.TimeOfDay).FirstOrDefault();

        while (times.Where(t => t <= min.AddMinutes(1)).Any())
        { 
            TheList.Add(min);
            Remove(times, min);
            min = times.OrderBy(c => c.Date).ThenBy(c => c.TimeOfDay).FirstOrDefault();
        }

        foreach (DateTime t in TheList)
            Console.WriteLine(t);
        Console.ReadKey();
    }

    static void Remove(List<DateTime> times, DateTime min)
    {
        times.RemoveAll(t => t <= min.AddMinutes(1));
    }

How can I take a list of dates, and remove any that are within 1 minute of another date in the list with LINQ?

3 Answers3