4

ModelCheckPoint gives options to save both for val_Acc and val_loss separately. I want to modify this in a way so that if val_acc is improving -> save model. if val_acc is equal to previous best val_acc then check for val_loss, if val_loss is less than previous best val_loss then save the model.

    if val_acc(epoch i)> best_val_acc:
        save model
    else if val_acc(epoch i) == best_val_acc:
        if val_loss(epoch i) < best_val_loss:
           save model
        else
           do not save model
Wojciech Wirzbicki
  • 3,887
  • 6
  • 36
  • 59
manv
  • 138
  • 1
  • 10

3 Answers3

11

You can just add two callbacks:

callbacks = [ModelCheckpoint(filepathAcc, monitor='val_acc', ...),
             ModelCheckpoint(filepathLoss, monitor='val_loss', ...)]

model.fit(......., callbacks=callbacks)

Using custom callbacks

You can do anything you want in a LambdaCallback(on_epoch_end=saveModel).

best_val_acc = 0
best_val_loss = sys.float_info.max 

def saveModel(epoch,logs):
    val_acc = logs['val_acc']
    val_loss = logs['val_loss']

    if val_acc > best_val_acc:
        best_val_acc = val_acc
        model.save(...)
    elif val_acc == best_val_acc:
        if val_loss < best_val_loss:
            best_val_loss=val_loss
            model.save(...)
    
callbacks = [LambdaCallback(on_epoch_end=saveModel)]

But this is nothing different from a single ModelCheckpoint with val_acc. You won't really be getting identical accuracies unless you're using very few samples, or you have a custom accuracy that doesn't vary much.

Innat
  • 16,113
  • 6
  • 53
  • 101
Daniel Möller
  • 84,878
  • 18
  • 192
  • 214
  • that should be it. – prosti Dec 12 '18 at 17:24
  • I have added two callbacks already(for now). But that wouldn't do what I want. Both callbacks will save weights based on either lowest loss or highest accuracy. But not according to my problem (as mentioned in question). – manv Dec 13 '18 at 02:26
  • That's perfect!! Thank you so much. Yes, I have less test samples that's why I am getting same accuracy again and again. @DanielMöller – manv Dec 14 '18 at 09:53
  • Also it needs a small modification. `best_val_loss` should be updated after first `if` .. `best_val_loss = val_loss` – manv Dec 14 '18 at 10:03
  • I didn't know this was possible. This is great! thanks @DanielMöller – Vincent Pakson Dec 20 '18 at 12:47
-1

You can actually check in their documentation!

to save you some time though, the callback, ModelCheckpoint accepts an argument called save_best_only which does what you want to happen, just set it to True. here's the link of the documentation

I misunderstood you're question. I guess if you want a more complex type of callback you could always use the base Callback function, which gives you more power since you could access both parmas and model. Check the docu out. You can start by testing it out and printing the params and determine which one you'd want to take note of.

Vincent Pakson
  • 1,891
  • 2
  • 8
  • 17
  • I am new to keras and trying to find out how to modify ModelCheckPoint class in keras. I tried searching on net questions regarding modelcheckpoint or if someone has modified this class. Couldn't get any success. If you have any reference please paste the link. Thanks – manv Dec 12 '18 at 07:07
  • As ive said. If you want more power for you're callback in building the model, it would be best for you to use the base `Callback` function. I added the link on the documentation. – Vincent Pakson Dec 12 '18 at 11:54
  • Thanks. It can be a way too for my problem. But I find LambdaCallback more easy. – manv Dec 14 '18 at 09:55
-1

Check out ModelCheckPoint in here. model.fit() method takes as a parameter the callback list. Make sure you have something like:

model.fit(..., callbacks=[mcp] ) where mcp = ModelCheckPoint() as defined.

Note: You may have multiple callbacks in the callback list.

For clarity I am adding some details but effectively this will do the same as model.save() function:

class ModelCheckpoint(Callback):
    """Save the model after every epoch.
    `filepath` can contain named formatting options,
    which will be filled the value of `epoch` and
    keys in `logs` (passed in `on_epoch_end`).
    For example: if `filepath` is `weights.{epoch:02d}-{val_loss:.2f}.hdf5`,
    then the model checkpoints will be saved with the epoch number and
    the validation loss in the filename.
    # Arguments
        filepath: string, path to save the model file.
        monitor: quantity to monitor.
        verbose: verbosity mode, 0 or 1.
        save_best_only: if `save_best_only=True`,
            the latest best model according to
            the quantity monitored will not be overwritten.
        mode: one of {auto, min, max}.
            If `save_best_only=True`, the decision
            to overwrite the current save file is made
            based on either the maximization or the
            minimization of the monitored quantity. For `val_acc`,
            this should be `max`, for `val_loss` this should
            be `min`, etc. In `auto` mode, the direction is
            automatically inferred from the name of the monitored quantity.
        save_weights_only: if True, then only the model's weights will be
            saved (`model.save_weights(filepath)`), else the full model
            is saved (`model.save(filepath)`).
        period: Interval (number of epochs) between checkpoints.
    """
prosti
  • 42,291
  • 14
  • 186
  • 151
  • I understand ModelCheckPoint class. And I know that one can give multiple callbacks in model.fit(). I want to judge model based on both val_acc and val_loss and then save accordingly. – manv Dec 12 '18 at 09:40