My X_test are 128x128x3 images and my Y_test are 512x512x3 images. I want to show, after each epoch, how the input (X_test) looked, how the expected output (Y_test) looked, but also how the actual output looked. So far, I've only figured out how to add the first 2 in Tensorboard. Here is the code that calls the Callback:
model.fit(X_train,
Y_train,
epochs=epochs,
verbose=2,
shuffle=False,
validation_data=(X_test, Y_test),
batch_size=batch_size,
callbacks=get_callbacks())
Here is the Callback's code:
import tensorflow as tf
from keras.callbacks import Callback
from keras.callbacks import TensorBoard
import io
from PIL import Image
from constants import batch_size
def get_callbacks():
tbCallBack = TensorBoard(log_dir='./logs',
histogram_freq=1,
write_graph=True,
write_images=True,
write_grads=True,
batch_size=batch_size)
tbi_callback = TensorBoardImage('Image test')
return [tbCallBack, tbi_callback]
def make_image(tensor):
"""
Convert an numpy representation image to Image protobuf.
Copied from https://github.com/lanpa/tensorboard-pytorch/
"""
height, width, channel = tensor.shape
print(tensor)
image = Image.fromarray(tensor.astype('uint8')) # TODO: maybe float ?
output = io.BytesIO()
image.save(output, format='JPEG')
image_string = output.getvalue()
output.close()
return tf.Summary.Image(height=height,
width=width,
colorspace=channel,
encoded_image_string=image_string)
class TensorBoardImage(Callback):
def __init__(self, tag):
super().__init__()
self.tag = tag
def on_epoch_end(self, epoch, logs={}):
# Load image
img_input = self.validation_data[0][0] # X_train
img_valid = self.validation_data[1][0] # Y_train
print(self.validation_data[0].shape) # (8, 128, 128, 3)
print(self.validation_data[1].shape) # (8, 512, 512, 3)
image = make_image(img_input)
summary = tf.Summary(value=[tf.Summary.Value(tag=self.tag, image=image)])
writer = tf.summary.FileWriter('./logs')
writer.add_summary(summary, epoch)
writer.close()
image = make_image(img_valid)
summary = tf.Summary(value=[tf.Summary.Value(tag=self.tag, image=image)])
writer = tf.summary.FileWriter('./logs')
writer.add_summary(summary, epoch)
writer.close()
return
I'm wondering where/how I can get the actual output of the network.
Another issue I'm having is that here is a sample of one of the images that is being ported into TensorBoard:
[[[0.10909907 0.09341043 0.08224604]
[0.11599099 0.09922747 0.09138277]
[0.15596421 0.13087936 0.11472746]
...
[0.87589591 0.72773653 0.69428956]
[0.87006552 0.7218123 0.68836991]
[0.87054225 0.72794635 0.6967475 ]]
...
[[0.26142332 0.16216267 0.10314116]
[0.31526875 0.18743924 0.12351286]
[0.5499796 0.35461449 0.24772873]
...
[0.80937942 0.62956016 0.53784871]
[0.80906054 0.62843601 0.5368183 ]
[0.81046278 0.62453899 0.53849678]]]
Is that the reason why my image = Image.fromarray(tensor.astype('uint8')) line might be generating images that do not look at all like the actual output? Here is a sample from TensorBoard:
I did try .astype('float64') but it launched an error because it is apparently not a type that is supported.
Anyhow, I'm unsure this really is the problem since the rest of my displayed images in the TensorBoard are all just white/gray/black squares (this one right there, conv2D_7, is actually the very last layer of my network and should thus display the actual images that are outputted, no?):
Ultimately, I would like something like this, which I'm already displaying after the training through matplot:
Finally, I would like to adress the fact that this callback is taking a long time to process. Is there a more efficient way to do that? It almost doubles my training time (probably because it needs to convert the numpy into images before saving them in the TensorBoard Log file).
