Calculate a moving Standard Deviation

Question

i found the following code snippet here on stackoverflow, but i have the problem that the stdev becomes NaN. Any ideas how to fix this?

public static void AddBollingerBands(ref SortedList<DateTime, Dictionary<string, double>> data, int period, int factor)
{
    double total_average = 0;
    double total_squares = 0;

    for (int i = 0; i < data.Count(); i++)
    {
        total_average += data.Values[i]["close"];
        total_squares += Math.Pow(data.Values[i]["close"], 2);

        if (i >= period - 1)
        {
            double total_bollinger = 0;
            double average = total_average / period;

            double stdev = Math.Sqrt((total_squares - Math.Pow(total_average,2)/period) / period);
            data.Values[i]["bollinger_average"] = average;
            data.Values[i]["bollinger_top"] = average + factor * stdev;
            data.Values[i]["bollinger_bottom"] = average - factor * stdev;

            total_average -= data.Values[i - period + 1]["close"];
            total_squares -= Math.Pow(data.Values[i - period + 1]["close"], 2);
        }
    }
}

Are you sure the `period` is never 0? Troubleshoot this a bit yourself. At what point does it become NaN in your loop (i.e. after how many iterations). Is it positive or negative infinity? Have any of the other values used in the calculation at the line `var stdev = ...` become NaN at that point? — Alex, Mar 25 '15 at 20:22
It seems to me that what you need is not the answer to this particular question, but rather to spend some time and learn to debug your code. Instrument the code and work out which operation leads to the problem? Narrow the problem down? — David Heffernan, Mar 25 '15 at 20:39
thanks, but after a few iterations the parameters have the following values: total_squares = 18.42483; total_average = 19.19628; period = 20; which ends up in a NaN — endeffects, Mar 25 '15 at 21:03
using those numbers: `total_squares - Math.Pow(total_average,2)/period` is `-0.00002829192`. I.e. a negative number that you are taking the square root of. That will result in NaN. — Alex, Mar 25 '15 at 21:06
so the problem belongs to the last code line? total_squares -= Math.Pow(data.Values[i - period + 1]["close"], 2); — endeffects, Mar 25 '15 at 21:12
Just as an aside - it appears that the `ref` in the parameter list is unnecessary. — Enigmativity, Mar 26 '15 at 00:54
This code also has an error when `period` > `data.Count` - The average is then wrong. — Enigmativity, Mar 26 '15 at 00:58

Alex · Accepted Answer · 2015-03-26T01:04:03.997

A numerically more stable variant is preferable when doing incremental / moving average and standard deviation calculations. One way to do this is using Knuth's algorithm, as shown in the code block below:

public class MovingAverageCalculator
{
    public MovingAverageCalculator(int period)
    {
        _period = period;
        _window = new double[period];
    }

    public double Average 
    {
        get { return _average; }
    }

    public double StandardDeviation
    {
        get 
        {
            var variance = Variance;
            if (variance >= double.Epsilon)
            {
                var sd = Math.Sqrt(variance);
                return double.IsNaN(sd) ? 0.0 : sd;
            }
            return 0.0;
        }
    }

    public double Variance
    {
        get 
        { 
            var n = N;
            return n > 1 ? _variance_sum / (n - 1) : 0.0; 
        }
    }

    public bool HasFullPeriod
    {
        get { return _num_added >= _period; }
    }

    public IEnumerable<double> Observations
    {
        get { return _window.Take(N); }
    }

    public int N
    {
        get { return Math.Min(_num_added, _period); }
    }

    public void AddObservation(double observation)
    {
        // Window is treated as a circular buffer.
        var ndx = _num_added % _period;
        var old = _window[ndx];     // get value to remove from window
        _window[ndx] = observation; // add new observation in its place.
        _num_added++;

        // Update average and standard deviation using deltas
        var old_avg = _average;
        if (_num_added <= _period)
        {
            var delta = observation - old_avg;
            _average += delta / _num_added;
            _variance_sum += (delta * (observation - _average));
        } 
        else // use delta vs removed observation.
        {
            var delta = observation - old;
            _average += delta / _period;
            _variance_sum += (delta * ((observation - _average) + (old - old_avg)));
        }
    }

    private readonly int _period;
    private readonly double[] _window;
    private int _num_added;
    private double _average;
    private double _variance_sum;
}

You could then use it in the following manner in your code example:

public static void AddBollingerBands(ref SortedList<DateTime, Dictionary<string, double>> data, int period, int factor)
{
    var moving_avg = new MovingAverageCalculator(period);
    for (int i = 0; i < data.Count(); i++)
    {
        moving_avg.AddObservation(data.Values[i]["close"]);
        if (moving_avg.HasFullPeriod)
        {
            var average = moving_avg.Average;
            var limit = factor * moving_avg.StandardDeviation;
            data.Values[i]["bollinger_average"] = average;
            data.Values[i]["bollinger_top"] = average + limit;
            data.Values[i]["bollinger_bottom"] = average - limit;
        }
    }
}

@endeffects glad this helped. A word of advice: make sure you put in some effort to find out how this site and community works. What to do when asking questions or when you come across questions or answers by other people that are helpful to you. — Alex, Mar 26 '15 at 16:37

score -1 · Answer 2 · answered Mar 25 '15 at 20:27

For stdev to become NaN something has to be going wrong in this assignment:

double stdev = Math.Sqrt((total_squares - Math.Pow(total_average,2)/period) / period);

You can't divide by zero, so make sure period isn't set to that. Easiest way to solve is to print out every variable before this line is called and see if something is already NaN or is mathematically unusable

Calculate a moving Standard Deviation

2 Answers2

Linked