How to display the average of multiple runs on tensorboard

Question

Is there a way to display the average of multiple different runs on tensorflow? I can only see them on the same graph (by sending the path of the different runs), but I want to see their average on the graph

Answer 1

As @dga mentioned this is not implemented yet. Here is some code that uses EventAccumulator to combine scalar tensorflow summary values. This can be extended to accommodate the other summary types.

import os
from collections import defaultdict

import numpy as np
import tensorflow as tf
from tensorboard.backend.event_processing.event_accumulator import EventAccumulator


def tabulate_events(dpath):

    summary_iterators = [EventAccumulator(os.path.join(dpath, dname)).Reload() for dname in os.listdir(dpath)]

    tags = summary_iterators[0].Tags()['scalars']

    for it in summary_iterators:
        assert it.Tags()['scalars'] == tags

    out = defaultdict(list)

    for tag in tags:
        for events in zip(*[acc.Scalars(tag) for acc in summary_iterators]):
            assert len(set(e.step for e in events)) == 1

            out[tag].append([e.value for e in events])

    return out


def write_combined_events(dpath, d_combined, dname='combined'):

    fpath = os.path.join(dpath, dname)
    writer = tf.summary.FileWriter(fpath)

    tags, values = zip(*d_combined.items())

    timestep_mean = np.array(values).mean(axis=-1)

    for tag, means in zip(tags, timestep_mean):
        for i, mean in enumerate(means):
            summary = tf.Summary(value=[tf.Summary.Value(tag=tag, simple_value=mean)])
            writer.add_summary(summary, global_step=i)

        writer.flush()

dpath = '/path/to/root/directory'

d = tabulate_events(dpath)

write_combined_events(dpath, d)

This solution assumes a directory structure like the following:

dpath
├── 1
│   └── events.out.tfevents.1518552132.Alexs-MacBook-Pro-2.local
├── 11
│   └── events.out.tfevents.1518552180.Alexs-MacBook-Pro-2.local
├── 21
│   └── events.out.tfevents.1518552224.Alexs-MacBook-Pro-2.local
├── 31
│   └── events.out.tfevents.1518552264.Alexs-MacBook-Pro-2.local
└── 41
    └── events.out.tfevents.1518552304.Alexs-MacBook-Pro-2.local

Answer 2

Please follow issue 376 to see progress on this. It's an active feature request with some progress in the last month, but as of now, there's not a way to do what you want. Yet.

Answer 3

Since there is still no build in functionality to do this I released a tool for that:

https://github.com/Spenhouet/tensorboard-aggregator

This tool can aggregate multiple tensorboard runs by their max, min, mean, median and standard deviation. The aggregates are either saved in new tensorboard summaries or as .csv files.

Answer 4

I created TensorBoard Reducer to do this with PyTorch.

pip install tensorboard-reducer
tb-reducer runs/of-your-model* -o output-dir -r mean,std,min,max

The aggregation results can be saved to disk either as new TensorBoard logs or CSV / JSON / Excel files.