WPF DynamicDataDisplay - Slow plotting with Markers

Question

I am having a really hard time waiting for the ChartPlotter in D3 to show itself, when using markers. Of course I am trying to plot a Gazillion records (well, 700,000 records). When using just a line, all is well (20 seconds or so). When using markers, we're talking 5 minutes. That's not acceptable.

Any ideas?

Here's what I have done, with explanations under it.

public static string MakeSimplePlot(double[][] xData, double[][] yData, string[] legend, string xAxisTitle, string yAxisTitle, bool[] showLines, bool[] showMarkers)
    {
        ChartPlotter plotter = new ChartPlotter();

        plotter.MainHorizontalAxis = new HorizontalAxis();
        plotter.MainVerticalAxis = new VerticalAxis();

        HorizontalAxisTitle horizontalAxisTitle = new HorizontalAxisTitle();
        horizontalAxisTitle.Content = xAxisTitle;
        plotter.AddChild(horizontalAxisTitle);

        VerticalAxisTitle verticalAxisTitle = new VerticalAxisTitle();
        verticalAxisTitle.Content = yAxisTitle;
        plotter.AddChild(verticalAxisTitle);

        Color[] plotColors = new Color[13] { Colors.Blue, Colors.Red, Colors.Green, Colors.Chartreuse, Colors.Yellow, Colors.Violet, Colors.Tan, Colors.Silver, Colors.Salmon, Colors.Lime, Colors.Brown, Colors.Chartreuse, Colors.DarkGray };

        for (int seriesCounter = 0; seriesCounter < legend.Count(); seriesCounter++)
        {
            DataFile clearedInputs = ClearExcess(new DataFile(xData[seriesCounter], yData[seriesCounter]));
            xData[seriesCounter] = clearedInputs.time;
            yData[seriesCounter] = clearedInputs.data;

            var xDataSource = new EnumerableDataSource<double>(xData[seriesCounter]);
            xDataSource.SetXMapping(x => x);

            var yDataSource = new EnumerableDataSource<double>(yData[seriesCounter]);
            yDataSource.SetYMapping(x => x);

            CompositeDataSource plotSeries = new CompositeDataSource(xDataSource, yDataSource);

            CirclePointMarker circlePointMarker = new CirclePointMarker();
            circlePointMarker.Fill = new SolidColorBrush(plotColors[seriesCounter]);
            circlePointMarker.Pen = new Pen(circlePointMarker.Fill, 0);

            circlePointMarker.Size = (showMarkers[seriesCounter] == false) ? 0 : 8;
            int lineWidth = (showLines[seriesCounter] == false) ? 0 : 2;

            if (showMarkers[seriesCounter] == false)
            {
                plotter.AddLineGraph(plotSeries, new Pen(circlePointMarker.Fill, lineWidth), new PenDescription("Dummy"));
            }
            else
            {
                plotter.AddLineGraph(plotSeries, new Pen(circlePointMarker.Fill, lineWidth), circlePointMarker, new PenDescription("Dummy"));
            }
        }

        UIParameters.plotWindow.mainGrid.Children.Clear();
        UIParameters.plotWindow.mainGrid.RowDefinitions.Clear();
        UIParameters.plotWindow.mainGrid.Children.Add(plotter);
        plotter.Viewport.FitToView();

        plotter.LegendVisible = false;
        plotter.NewLegendVisible = false;            

        if (legend.Count() > 1)
        {
            ShowLegend(legend, plotColors);
        }

        UIParameters.plotWindow.WindowState = WindowState.Minimized;
        UIParameters.plotWindow.WindowState = WindowState.Normal;

        string filename = Path.ChangeExtension(Path.GetTempFileName(), "png");

        RenderTargetBitmap targetBitmap = new RenderTargetBitmap((int)UIParameters.plotWindow.mainGrid.ActualWidth, (int)UIParameters.plotWindow.mainGrid.ActualHeight, 96d, 96d, PixelFormats.Default);
        targetBitmap.Render(UIParameters.plotWindow.mainGrid);
        PngBitmapEncoder encoder = new PngBitmapEncoder();
        encoder.Frames.Add(BitmapFrame.Create(targetBitmap));
        using (var fileStream = File.Open(filename, FileMode.OpenOrCreate))
        {
            encoder.Save(fileStream);
            UIParameters.plotWindow.mainGrid.Clip = null;

            GC.Collect();
            GC.WaitForPendingFinalizers();
            GC.Collect();

            targetBitmap.Freeze();

            if (targetBitmap != null) targetBitmap.Clear();
            targetBitmap = null;
            GC.Collect();
            GC.WaitForPendingFinalizers();

            GC.Collect();
            GC.WaitForPendingFinalizers();
        }

        return filename;
    }

Explanations:

1. I hide the plotter legend and make my own using ShowLegend, since the legend does not show if it has only markers (am I wrong?)
2. I minimize and maximize the plot window, since otherwise the plot does not update, or it updates but does not get saved to a file. This also works if I move the window (I guess some kind of redraw event), but since the process is autoamatic, the user does not have any interaction. I tries Invalidate, to no avail. Ideas?

Thanks!

Answer 1

I wrote my own class to hide markers when they are off screen. It's a virtualization technique that speeds up performance tenfold when you don't have tons of markers on screen. It looks like this :

using System;
using System.Windows;
using System.Windows.Media;
using Microsoft.Research.DynamicDataDisplay.DataSources;
using Microsoft.Research.DynamicDataDisplay.PointMarkers;
using Microsoft.Research.DynamicDataDisplay.Common;

namespace Microsoft.Research.DynamicDataDisplay.Charts {
public class FilteredMarkerPointsGraph : MarkerPointsGraph {
    public FilteredMarkerPointsGraph()
        : base() {
        ;
    }

    public FilteredMarkerPointsGraph(IPointDataSource dataSource)
        : base(dataSource) {
        ;
    }

    protected override void OnRenderCore(DrawingContext dc, RenderState state) {
        // base.OnRenderCore
        if (DataSource == null) return;
        if (Marker == null) return;

        var left = Viewport.Visible.Location.X;
        var right = Viewport.Visible.Location.X + Viewport.Visible.Size.Width;
        var top = Viewport.Visible.Location.Y;
        var bottom = Viewport.Visible.Location.Y + Viewport.Visible.Size.Height;

        var transform = Plotter2D.Viewport.Transform;

        DataRect bounds = DataRect.Empty;
        using (IPointEnumerator enumerator = DataSource.GetEnumerator(GetContext())) {
            Point point = new Point();
            while (enumerator.MoveNext()) {
                enumerator.GetCurrent(ref point);                                       

                if (point.X >= left && point.X <= right && point.Y >= top && point.Y <= bottom)
                {
                    enumerator.ApplyMappings(Marker);

                    Point screenPoint = point.DataToScreen(transform);

                    bounds = DataRect.Union(bounds, point);
                    Marker.Render(dc, screenPoint);
                }
            }
        }

        Viewport2D.SetContentBounds(this, bounds);
    }
}

Make sure you call FilteredMarkerPointsGraph in the XAML instead of MarkerPointsGraph!

EDIT

1. I'm not sure what you need with the legend with markers, I've not actually used a legend in any of my graphs, but your solution seems to be fine.

2. Redrawing the plot is quite easy actually.

The best way that I have found to do this, is to have a Property in your code behind that represents the DataSource and bind the chart's DataSource to that property. Have your code behind implement INotifyPropertyChanged and call OnPropertyChanged every time you update or re-assign your data source. This will force the plotter to observe the binding and redraw your graph.

Example:

EnumerableDataSource<Point> m_d3DataSource;
public EnumerableDataSource<Point> D3DataSource {
get {
    return m_d3DataSource;
}
set {                
    //you can set your mapping inside the set block as well             
    m_d3DataSource = value;
    OnPropertyChanged("D3DataSource");
}
}     

protected void OnPropertyChanged(PropertyChangedEventArgs e) {
    PropertyChangedEventHandler handler = PropertyChanged;
    if (handler != null) {
        handler(this, e);
    }
} 

protected void OnPropertyChanged(string propertyName) {
    OnPropertyChanged(new PropertyChangedEventArgs(propertyName));
}  

And about your performance with your markers.. It's hard to pinpoint exactly what would be causing your performance issue, but my recommendation is to try using a different data source. I've been using EnumerableDataSource and it's always worked like a charm. Try bringing in your data in a singular object and setting the mapping in your set block like as shown above using:

value.SetYMapping(k => Convert.ToDouble(k.yData));
value.SetXMapping(k => Convert.ToDouble(k.xData));

The only thing you have to worry about is the mapping in Enumerable data source and D3 should handle the rest for you.

Answer 2

Well user can't probably see markers anyway when you are displaying the "Gazillion" of points: can't you switch the mode from line to markers when the zoom level is more reasonable ?