What is the mathematics behind the "smoothing" parameter in TensorBoard's scalar graphs?

Question

I presume it is some kind of moving average, but the valid range is between 0 and 1.

bluesummers · Accepted Answer · 2023-03-13T22:32:52.743

91

ORIGINAL ANSWER

It is called exponential moving average, below is a code explanation how it is created.

Assuming all the real scalar values are in a list called scalars the smoothing is applied as follows:

def smooth(scalars: List[float], weight: float) -> List[float]:  # Weight between 0 and 1
    last = scalars[0]  # First value in the plot (first timestep)
    smoothed = list()
    for point in scalars:
        smoothed_val = last * weight + (1 - weight) * point  # Calculate smoothed value
        smoothed.append(smoothed_val)                        # Save it
        last = smoothed_val                                  # Anchor the last smoothed value
        
    return smoothed

UPDATED ANSWER

As @SaPropper correctly pointed out, TensorBoard now includes the debiasing factor.

edited Mar 13 '23 at 22:32

answered Mar 19 '18 at 07:06

bluesummers

11,365
8
72
108

2

I wish you could pick between that and a sliding moving average! – Avedis Dec 21 '20 at 16:11
This was really helpful! Thanks. – Abhik Banerjee Jan 13 '21 at 12:08
This is not the Python equivalent to the TensorBoard implementation – SaPropper Feb 24 '23 at 16:41
To be more explicit: the TensorBoard implementation currently also has a debiasing factor, as in SaPropper's answer. – Eric O. Lebigot Mar 13 '23 at 15:32

score 5 · Answer 2 · edited Mar 13 '23 at 14:38

Here is the actual piece of source code that performs that exponential smoothing with some additional de-biasing explained in the comments to compensate for the choice of the zero initial value:

last = last * smoothingWeight + (1 - smoothingWeight) * nextVal

Source: https://github.com/tensorflow/tensorboard/blob/34877f15153e1a2087316b9952c931807a122aa7/tensorboard/components/vz_line_chart2/line-chart.ts#L714

score 3 · Answer 3 · answered Feb 11 '23 at 17:16

The implementation of EMA smoothing used for TensorBoard can be found here.

The equivalent in Python is actually:

def smooth(scalars: list[float], weight: float) -> list[float]:
    """
    EMA implementation according to
    https://github.com/tensorflow/tensorboard/blob/34877f15153e1a2087316b9952c931807a122aa7/tensorboard/components/vz_line_chart2/line-chart.ts#L699
    """
    last = 0
    smoothed = []
    num_acc = 0
    for next_val in scalars:
        last = last * weight + (1 - weight) * next_val
        num_acc += 1
        # de-bias
        debias_weight = 1
        if weight != 1:
            debias_weight = 1 - math.pow(weight, num_acc)
        smoothed_val = last / debias_weight
        smoothed.append(smoothed_val)

    return smoothed

What is the mathematics behind the "smoothing" parameter in TensorBoard's scalar graphs?

3 Answers3

Linked