-2

I tried to use solution suggested in this answer: cross-correlation using mathdotnet but I always get maximum correlation at lag 0. My source data has definitely significant lags that happen once in a while. When I run this in Python using Pandas package, I am able to detect most of the places that have latency as strongest correlation will be at certain lags, but with Math.Net its always at lag 0 and the decreasing on every subsequent lag.

Below is a link to dataset I use resampled and normalized to 500ms averages. 500ms_0930_1000.csv

Here is a code for cross correlation borrowed from other question. I am running this code using a sliding window that takes 40 points (20 seconds) and slides by 20 (10 seconds) every step.

public IList<(double Corr, double Lag)> CalculateCrossCorrelation(double[] x1, double[] x2)
        {
            var len = x1.Length;
            var len2 = 2 * len;
            var len3 = 3 * len;
            var s1 = new double[len3];
            var s2 = new double[len3];

            Array.Copy(x1, 0, s1, len, len);
            Array.Copy(x2, 0, s2, 0, len);

            var results = new List<(double Corr, double Lag)>(len2);
            for (int i = 0; i < len2; i++)
            {
                results.Add((Corr: Correlation.Pearson(s1, s2), Lag: i - len));
                Array.Copy(s2, 0, s2, 1, s2.Length - 1);
                s2[0] = 0;
            }

            return results;
        }
Alex Michel
  • 416
  • 3
  • 13
  • Please explain what you want to do. – SQL Police Sep 23 '19 at 08:10
  • I want to determine lags between pairs of sources/3rd party APIs, in order to detect which of them has latency vs the other, so I can generate alerts based on this with resolution of few seconds. In dataset I provided its clear that one of the sources was subject to extreme latency of >10 seconds in few periods (e.g. if you create line chart in Excel from it it is clearly visible around 9:36 and 9:52-9:53) – Alex Michel Sep 23 '19 at 08:38

1 Answers1

0

Why are you setting s2[0]=0?

You don't do this on the initial pass. It gives an outl

phv3773
  • 487
  • 4
  • 10