Running model.fit() and TensorBoard at the same time?

Question

I have an interesting (for me) question about running model.fit() and tensorboard at the same time.

I did some research on internet about "Threading", "Processing", "Multi-processing", tried examples but couldn't solve my problem.

I want to run TensorBoard and model.fit() at the same time like:

from threading import Thread
import subprocess

def startTensorboard(log_dir):
    # Tried both
    os.system('tensorboard --logdir '+ log_dir)
    # subprocess.call(['tensorboard', '--logdir', log_dir])

tensorboard = tf.keras.callbacks.TensorBoard(log_dir='logs', histogram_freq=0,
                          write_graph=True, write_images=False)

Thread(target = startTensorboard('logs')).start()

Thread(target = model.fit_generator(
                self.train_data_gen,
                steps_per_epoch=self.STEPS_PER_EPOCH,
                validation_data = self.test_data_gen,
                validation_steps = self.VALID_STEPS_PER_EPOCH,
                epochs=self.epoch,
                callbacks=[tensorboard])).start()

Is that possible? When I ran this code, TensorBoard is running but model.fit() isn't working.

score 1 · Accepted Answer · answered Dec 25 '19 at 05:22

The following is a working example that I think does what you want. I'm using Process from the multiprocessing module. Note that it seems to be important that the model is defined inside the function that you set as the target for Process when you're calling the fit function, as indicated in this post. I tried defining the model outside the function call and it would initialize the model, but then the training would just hang indefinitely.

When I run this on my laptop, tensorboard takes a little bit to get started, but usually by the time training hits epoch 70 tensorboard has launched and it continues to run until you kill it with Ctrl+C.

import os
import numpy as np
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
from multiprocessing import Process


def startTensorboard(logdir):
    # Start tensorboard with system call
    os.system("tensorboard --logdir {}".format(logdir))


def fitModel():
    # Create your model
    model = Sequential()
    model.add(Dense(32, activation='relu', input_dim=100))
    model.add(Dense(1, activation='sigmoid'))
    model.compile(optimizer='rmsprop',
                  loss='binary_crossentropy',
                  metrics=['accuracy'])

    # Some mock training data
    data = np.random.random((1000, 100))
    labels = np.random.randint(2, size=(1000, 1))

    # Run the fit function
    model.fit(data, labels, epochs=100, batch_size=32)


if __name__ == '__main__':
    # Run both processes simultaneously
    Process(target=startTensorboard, args=("logs",)).start()
    Process(target=fitModel).start()

Running model.fit() and TensorBoard at the same time?

1 Answers1