2

I provide both training and validation data as Dataset iterators during training by fit function. I want to save best model by validation dataset accuracy. However, during training at each epoch, I get warning like Can save best model only with val_acc, skipping.

I compiled the model with metrics = ['accuracy'], declared monitor='val_acc' in callback, and provided validation data as Dataset iterator like validation_data=my_val_dataset in fit. What else am I missing here?

I would really appreciate any help.

My Jupyter notebook is in my GitHub repo.

Rick Blaine
  • 175
  • 1
  • 6

1 Answers1

3

Make sure val_acc exists in the logs.

Get the history from a short training: 

histCallback = model.fit(smallBatch_X, smallBatch_Y, epochs = 1)

Check what is in it:

for key in histCallback.history:
    print(key)

Maybe you will find val_accuracy instead. Or maybe you have multiple outputs and will find val_output_2_acc or something in a similar style.

Daniel Möller
  • 84,878
  • 18
  • 192
  • 214
  • Yes I found `val_accuracy` as well as `accuracy`. In `histCallback.history` I see array of these values per each epoch. Then in `./training_checkpoints/ckpt_1/` I see one `saved_model`. I have no idea if that's the best model or the model from the last epoch. What does this mean? My revised output per your suggestion is in https://github.com/shinchan75034/testrep/blob/master/tf20_dist_train.ipynb – Rick Blaine Sep 24 '19 at 04:48
  • @RickBlaine This means that you need `monitor='val_accuracy'`. The value `val_acc` doesn't exist. – Daniel Möller Sep 24 '19 at 13:16
  • great! Thank you @DanielMöller for the trick to look up proper names for these output arrays. Now I got it to work. I see output like this which shows at this epoch, model assets are saved because validation accuracy is best : `Train for 5 steps, validate for 5 steps Epoch 1/10 1/5 [=====>........................] - ETA: 13s - loss: 0.4088 - accuracy: 0.8750INFO:tensorflow:Assets written to: ./training_checkpoints/ckpt_1/assets` – Rick Blaine Sep 24 '19 at 16:08
  • Ok :) -- You can disable the message, if you like, with `verbose=0`. -- If you consider that this answers your question, please mark it as answered. – Daniel Möller Sep 24 '19 at 16:13