How to use Keras TensorBoard callback for grid search

Question

I'm using the Keras TensorBoard callback. I would like to run a grid search and visualize the results of each single model in the tensor board. The problem is that all results of the different runs are merged together and the loss plot is a mess like this:

How can I rename each run to have something similar to this:

Here the code of the grid search:

df = pd.read_csv('data/prepared_example.csv')

df = time_series.create_index(df, datetime_index='DATE', other_index_list=['ITEM', 'AREA'])

target = ['D']
attributes = ['S', 'C', 'D-10','D-9', 'D-8', 'D-7', 'D-6', 'D-5', 'D-4',
       'D-3', 'D-2', 'D-1']

input_dim = len(attributes)
output_dim = len(target)

x = df[attributes]
y = df[target]

param_grid = {'epochs': [10, 20, 50],
              'batch_size': [10],
              'neurons': [[10, 10, 10]],
              'dropout': [[0.0, 0.0], [0.2, 0.2]],
              'lr': [0.1]}

estimator = KerasRegressor(build_fn=create_3_layers_model,
                           input_dim=input_dim, output_dim=output_dim)


tbCallBack = TensorBoard(log_dir='./Graph', histogram_freq=0, write_graph=True, write_images=False)

grid = GridSearchCV(estimator=estimator, param_grid=param_grid, n_jobs=-1, scoring=bug_fix_score,
                            cv=3, verbose=0, fit_params={'callbacks': [tbCallBack]})

grid_result = grid.fit(x.as_matrix(), y.as_matrix())

score 5 · Accepted Answer · edited Aug 02 '17 at 17:07

I don't think there is any way to pass a "per-run" parameter to GridSearchCV. Maybe the easiest approach would be to subclass KerasRegressor to do what you want.

class KerasRegressorTB(KerasRegressor):

    def __init__(self, *args, **kwargs):
        super(KerasRegressorTB, self).__init__(*args, **kwargs)

    def fit(self, x, y, log_dir=None, **kwargs):
        cbs = None
        if log_dir is not None:
            params = self.get_params()
            conf = ",".join("{}={}".format(k, params[k])
                            for k in sorted(params))
            conf_dir = os.path.join(log_dir, conf)
            cbs = [TensorBoard(log_dir=conf_dir, histogram_freq=0,
                               write_graph=True, write_images=False)]
        super(KerasRegressorTB, self).fit(x, y, callbacks=cbs, **kwargs)

You would use it like:

# ...

estimator = KerasRegressorTB(build_fn=create_3_layers_model,
                             input_dim=input_dim, output_dim=output_dim)

#...

grid = GridSearchCV(estimator=estimator, param_grid=param_grid,
n_jobs=1, scoring=bug_fix_score,
                  cv=2, verbose=0, fit_params={'log_dir': './Graph'})

grid_result = grid.fit(x.as_matrix(), y.as_matrix())

Update:

Since GridSearchCV runs the same model (i.e. the same configuration of parameters) more than once due to cross-validation, the previous code will end up putting multiple traces in each run. Looking at the source (here and here), there doesn't seem to be a way to retrieve the "current split id". At the same time, you shouldn't just check for existing folders and add subfixes as needed, because the jobs run (potentially at least, although I'm not sure if that's the case with Keras/TF) in parallel. You can try something like this:

import itertools
import os

class KerasRegressorTB(KerasRegressor):

    def __init__(self, *args, **kwargs):
        super(KerasRegressorTB, self).__init__(*args, **kwargs)

    def fit(self, x, y, log_dir=None, **kwargs):
        cbs = None
        if log_dir is not None:
            # Make sure the base log directory exists
            try:
                os.makedirs(log_dir)
            except OSError:
                pass
            params = self.get_params()
            conf = ",".join("{}={}".format(k, params[k])
                            for k in sorted(params))
            conf_dir_base = os.path.join(log_dir, conf)
            # Find a new directory to place the logs
            for i in itertools.count():
                try:
                    conf_dir = "{}_split-{}".format(conf_dir_base, i)
                    os.makedirs(conf_dir)
                    break
                except OSError:
                    pass
            cbs = [TensorBoard(log_dir=conf_dir, histogram_freq=0,
                               write_graph=True, write_images=False)]
        super(KerasRegressorTB, self).fit(x, y, callbacks=cbs, **kwargs)

I'm using os calls for Python 2 compatibility, but if you are using Python 3 you may consider the nicer pathlib module for path and directory handling.

Note: I forgot to mention it earlier, but just in case, note that passing write_graph=True will log a graph per run, which, depending on your model, could mean a lot (relatively speaking) of this space. The same would apply to write_images, although I don't know the space that feature requires.

Thank you for the detailed suggestion. I'll try it later today and I'll let you know. Just one consideration: Does this solution create several folders? In that case am I able to display all the runs in a single tensorboard or I have to run several instances of it? — paolof89, Aug 02 '17 at 10:36
@paolof89 Yes, it does create a directory per experiment, but, in fact, the "Runs" that you see in TensorBoard are really just subfolders with log information. If you open TensorBoard in the root of the logs (in the example `./Graph`) you will see one "run" per experiment, all of them together, or you can open TensorBoard in the directory of a specific run to take a closer look. — jdehesa, Aug 02 '17 at 10:50
I tested it, it works but there is one last issue. The GridSearchCV implements a k-fold tecnique, so in every folder you find k graph. The minimum k-fold value is 2 so my problem is not yet solved. Any idea about it? — paolof89, Aug 02 '17 at 15:10
I'll open an issue on keras github in case some user had in the same issue — paolof89, Aug 02 '17 at 15:32
@paolof89 I've updated the answer with a possible solution to that. — jdehesa, Aug 02 '17 at 15:42

score 2 · Answer 2 · edited Apr 08 '19 at 08:56

It's easy, just save logs to separate dirs with concatenated parameters string as dir name:

Here is example using date as name of run:

from datetime import datetime

datetime_str = ('{date:%Y-%m-%d-%H:%M:%S}'.format(date=datetime.now()))
callbacks = [
    ModelCheckpoint(model_filepath, monitor='val_loss', save_best_only=True, verbose=0),
    TensorBoard(log_dir='./logs/'+datetime_str, histogram_freq=0, write_graph=True, write_images=True),
]

history = model.fit_generator(
    generator=generator.batch_generator(is_train=True),
    epochs=config.N_EPOCHS,
    steps_per_epoch=100,
    validation_data=generator.batch_generator(is_train=False),
    validation_steps=10,
    verbose=1,
    shuffle=False,
    callbacks=callbacks)

One can also use `datetime_str = datetime.now().strftime("%Y-%m-%d-%H-%M-%S")`. Based on: http://strftime.org , one can customize his own time string. — swiss_knight, Apr 08 '19 at 09:20

How to use Keras TensorBoard callback for grid search

2 Answers2

Linked