0

I have a generic class for performing some filtering actions on ObservableCollections - a small custom pagination mechanism.

public class Paginate<T> : INotifyPropertyChanged
{
    public readonly ObservableCollection<T> all_data; 
    public ObservableCollection<T> filtered_data;
   
    public Paginate(ObservableCollection<T> _data)
    {
        all_data = _data;
        filtered_data = all_data;
    }

    public ObservableCollection<T> Filter(Func<T, bool> obj)
    {
        //Filter collection
            
        //This takes about 10ms to finish - no errors at all
        //var test = all_data.Where(obj).ToList(); 

        //This doesn't finish at all, consuming a lot of memory 
        filtered_data = all_data.Where(obj).ToObservableCollection();
 
        //...

        return filtered_data;
    }

}

My problem is in Filter method, converting IEnumerable to ObservableCollection doesn't finish. However, when Linq result are small number of records It does finish correctly. Here is how I call everything in ViewModel (command for button):

private void Filter_Execute(object parameter)
{
      //Test_data is ObservableCollection<Student>, _paginate is instance of Paginate class
        Test_data = _paginate.Filter(Filter_data);
            
      //...
         
 }

public bool Filter_data(object obj)
{
     //Filtering logic      
     if (obj is Student student)
     {
          return ((string.IsNullOrEmpty(Name) ? true : student.NAME == Name)
                 & (string.IsNullOrEmpty(Surname) ? true : student.SURNAME == Surname)
                 & (string.IsNullOrEmpty(Age) ? true : student.AGE == Convert.ToInt16(Age))
                  );
      }
      return false;
 }

And my extension method for converting IEnumerable to ObservableCollection:

public static ObservableCollection<T> ToObservableCollection<T> (this IEnumerable<T> source)
{
    if (source == null)
    {
        throw new ArgumentNullException("source");
    }
    return new ObservableCollection<T>(source);
 }

So what am I doing wrong here ?

Update

My problem lies on UI side when binding to DataGrid. Obviously when Linq results exceeds number of items that you are displaying in UI you mustn't have property ScrollViewer.CanContentScroll set to False. This somehow causes huge memory leak that doesn't stop.

I'm working on a solution now.

dbc
  • 104,963
  • 20
  • 228
  • 340
Lucy82
  • 654
  • 2
  • 12
  • 32
  • (Aside: you probably want to be using `&&` instead of `&`) – canton7 May 11 '21 at 08:37
  • I can't see anything obvious -- it all looks fairly sensible. You need to find out where it's taking its time -- a profiler is the best way to do this, but randomly breaking the debugger and seeing what line is executing is a quick-and-dirty way to get a first feel – canton7 May 11 '21 at 08:38
  • Nothing obvious in the code you provided, just make sure you understand how IEnumerable(s) work. See https://stackoverflow.com/a/52450636/8695782 – Alexandru Clonțea May 11 '21 at 08:41
  • @canton7, I had done debugging. It stops in line **filtered_data = all_data.Where(obj).ToObservableCollection();**, and memory just keeps rising. – Lucy82 May 11 '21 at 08:41
  • Not on the line `return new ObservableCollection(source);` or a line inside `Filter_data`? That's very odd. – canton7 May 11 '21 at 08:42
  • @canton7, well basically here yes **return new ObservableCollection(source);**. – Lucy82 May 11 '21 at 08:44
  • So, which of those two lines is it stuck on? They have very different implications – canton7 May 11 '21 at 08:46
  • @AlexandruClonțea, If I understand completely, I should rather use `.ToList()` first ? – Lucy82 May 11 '21 at 08:46
  • @canton7, `filtered_data = all_data.Where(obj).ToObservableCollection();` calls extension method and should return ObservableCollection of same T type - but It doesn't, memory keeps rising rapidly. However, when linq result is small everything works as expected. – Lucy82 May 11 '21 at 08:49
  • Yes, I thought that that might help narrow it down, as I see you pointed out that works. Now that I see the other discussion you're having here, that may be unrelated. Are you "recreating" the observable on each call to Filter? Have you tried using the Rx built-in ToObservable extension method? – Alexandru Clonțea May 11 '21 at 08:51
  • @AlexandruClonțea, no never seen that, just heard something about It. What is this ? – Lucy82 May 11 '21 at 08:52
  • Disregard my last advice:) just reading http://introtorx.com/Content/v1.0.10621.0/04_CreatingObservableSequences.html makes sense. Especially "When transitioning from IEnumerable to IObservable, you should carefully consider what you are really trying to achieve. You should also carefully test and measure the performance impacts of your decisions. Consider that the blocking synchronous (pull) nature of IEnumerable sometimes just does not mix well with the asynchronous (push) nature of IObservable." – Alexandru Clonțea May 11 '21 at 08:55
  • 1
    There's no `IObservable` in this code -- that's a complete red herring. `ObservableCollection` is not in any way related to `IObservable` – canton7 May 11 '21 at 08:56
  • Oh my bad, I misread the code. You're right, ObservableCollection is a different beast :) – Alexandru Clonțea May 11 '21 at 08:57
  • @AlexandruClonțea, I think there is some catch while working with generics, but then why everything works when linq results are small ? Maybe some flaw in calling `Func obj˙` ? – Lucy82 May 11 '21 at 08:59
  • I don't think so, I'm trying to get a small debugging setup to see what's going on. When is this code called? Can you provide an 'equivalent" call that would cause this? (minimal repro) – Alexandru Clonțea May 11 '21 at 09:01
  • @AlexandruClonțea, It's all code there is up there. The only thing missing is Student Class, just a simple class with 3 properties. And ofcourse data of Students....Meanwhile I tried converting all linq results to `.ToList();˛` - and got same result. But this time It looks like a memory leak - Test_data is reinitialized with new data, but memory still rises rapidly. – Lucy82 May 11 '21 at 09:08
  • 1
    Is `all_data` definitely an `ObservableCollection` in your code? It smells a lot like a query provider is having to materialise all results from a database query in order to run that filter, and that's the thing that's taking time and memory. That would cause the symptoms you're seeing, but 1) That would also affect `.ToList()` as well as `.ToObservableCollection()`, and 2) that would only work if `all_data` was an `IQueryable` – canton7 May 11 '21 at 09:12
  • @canton7, yes it is, `ObservableCollection`. I fill this collection much before calling any other things. – Lucy82 May 11 '21 at 09:14
  • canton7 may be on to something.... I've tried doing this "from scratch", piecing together the code that was provided and was unable to repro this issue.(https://dotnetfiddle.net/el4f5j) Tried on my machine with up to 5 mil records, still runs - memory usage does seem a bit high. – Alexandru Clonțea May 11 '21 at 09:25
  • I think we need a [mcve] – canton7 May 11 '21 at 09:32
  • @canton7, I may be on something too. Maybe I overlooked everything, and problem might be a WPF binding issue. This collection is intended for binding to WPF DataGrid. When I removed collection from It's ItemsSource, I get no memory leaks ! Still a strange issue though. – Lucy82 May 11 '21 at 09:32
  • 50 mil items test done. Takes 10Gb of memory, but no crash =)) – Alexandru Clonțea May 11 '21 at 09:33
  • 1
    That *shouldn't* make any difference -- `filtered_data` is a field, not a property, so you can't be binding to it. The earliest point at which it could be bound to is after `Filter` returns, which is after you're telling us this problem happens – canton7 May 11 '21 at 09:34
  • Is ToObservableCollection only called once? – Alexandru Clonțea May 11 '21 at 09:36
  • @canton7, I found my problem - but I didn't get this one coming !!... `filtered data` is not a problem at all, or any of properties such as `Test_data` - which finally binds to DataGrid. Problem was `` in my DataGrid style. This obviously blocks collection in UI to redraw. But how to disable scrolling if I want to paginate my DataGrid ?? – Lucy82 May 11 '21 at 09:38
  • There isn't enough information for me to begin to comment I'm afraid – canton7 May 11 '21 at 09:40
  • @canton7, as I see It, If you set new binding to DataGrid ItemsSource and you set `False` to `CanContenScroll` then things get messed up - fi your records exceed number of items your DataGrid present in UI. So my answer is probably to set ContentScroll to false after re-initialsing collection. – Lucy82 May 11 '21 at 09:43

0 Answers0