0

I am training a fairly large keras unet model with some obscure data to do semantic image segmentation. The val_loss decreases initially but after some hours it starts to shoot up. It was likely overfitting and I put in more data only to delay the same behavior.

I want to observe what is going on in the validation stage. Is it possible to output a "per sample accuracy and loss" after each epoch since val_loss and val_accuracy is computed after each epoch not batch?

To put it another way, I want to find out the sample (let's assume image or numpy array) that gives the highest accuracy and the lowest accuracy and visualize them. I already have written a custom data generator from which I can output the file/array name/number that is being processed but what I am looking to do has been explained above.

Quite likely, I need some sort of a callback.

Clarification (after comment): by sample I mean a datapoint (or an image in this context) .

The problem at hand is a semantic segmentation problem (with three classes) not a classification one.

So basically, on a high level, I am trying to ask the question: in my validation set, which image is being quantified to have high accuracy/low loss, so I can do a sanity check to see if the functions are making sense?

CellDamag3
  • 1
  • 2
  • By *per sample* you actually mean *per epoch*, right? Please edit & update your question to confirm & clarify this (*sample* means something different). – desertnaut Dec 03 '20 at 22:17
  • Hi there. By *sample* I meant a *datapoint* (or an *image* in this context) . Updating the clarification, Appreciate the help. – CellDamag3 Dec 04 '20 at 20:17
  • So, *per sample accuracy* and *the sample that gives the highest/lowest accuracy* do not make any sense at all; accuracy is the percentage of points correctly classified, and it is meaningless for any single sample. You can, however request these samples regarding the *loss*; own answer in [Loss & accuracy - Are these reasonable learning curves?](https://stackoverflow.com/questions/47817424/loss-accuracy-are-these-reasonable-learning-curves/47819022#47819022) may help you clarify the issue. – desertnaut Dec 04 '20 at 22:42
  • @desertnaut : Ahh, I see the what confusion here. The problem at hand is a semantic segmentation problem (with three classes) not a classification one. So basically, on a high level, I am asking, in my validation set, which image is being quantified to have high accuracy so I can do a sanity check to see if the functions are making sense. Loss works too. Appreciate the link. Thanks. – CellDamag3 Dec 05 '20 at 10:44
  • Maybe clarify this explicitly in your post, then, instead of the comments? – desertnaut Dec 05 '20 at 11:00
  • 1
    Yes, thank you. Very much appreciate you taking the time. I edited the post. Again, your reply in the link really helped. – CellDamag3 Dec 05 '20 at 23:31

1 Answers1

0

This may be possible without a custom callback using Tensorboard with the update_freq option.

logdir = "./logs/" + datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = keras.callbacks.TensorBoard(log_dir=logdir, update_freq=
'batch')

model.fit(..., callbacks=[tensorboard_callback])

And open TensorBoard with

%tensorboard --logdir logs/scalars
Josh Ziegler
  • 436
  • 4
  • 9