keras extract best val loss

Question

The network is like

inp=Input((1,12))
dense0=GRU(200,activation='relu',recurrent_dropout=0.2,return_sequences=True)(inp)
drop0=Dropout(0.3)(dense1)
dense1=GRU(200,activation='relu',recurrent_dropout=0.2)(drop0)
drop1=Dropout(0.3)(dense1)
dense1=Dense(200,activation='relu')(inp)
drop1=Dropout(0.3)(dense1)
dense2=Dense(200,activation='relu')(drop1)
drop2=Dropout(0.3)(dense2)
dense3=Dense(100,activation='relu')(drop2)
drop3=Dropout(0.3)(dense3)
out=Dense(6,activation='relu')(drop2)

md=Model(inputs=inp,outputs=out)
##md.summary()
opt=keras.optimizers.rmsprop(lr=0.000005)
md.compile(opt,loss='mean_squared_error')
esp=EarlyStopping(patience=90, verbose=1, mode='auto')
md.fit(x_train.reshape((8105,1,12)),y_train.reshape((8105,1,6)),batch_size=2048,epochs=1500,callbacks=[esp], validation_split=0.2)

The output：

    Epoch 549/1500
    6484/6484 [==============================] - 0s 13us/step - loss: 0.0589 - val_loss: 0.0100
    Epoch 550/1500
    6484/6484 [==============================] - 0s 10us/step - loss: 0.0587 - val_loss: 0.0099
    Epoch 551/1500
    6484/6484 [==============================] - 0s 12us/step - loss: 0.0584 - val_loss: 0.0100
    Epoch 552/1500
    6484/6484 [==============================] - 0s 12us/step - loss: 0.0593 - val_loss: 0.0100
    Epoch 553/1500
    6484/6484 [==============================] - 0s 12us/step - loss: 0.0584 - val_loss: 0.0100
    Epoch 554/1500
    6484/6484 [==============================] - 0s 15us/step - loss: 0.0587 - val_loss: 0.0101
    Epoch 555/1500
    6484/6484 [==============================] - 0s 12us/step - loss: 0.0583 - val_loss: 0.0100
    Epoch 556/1500
    6484/6484 [==============================] - 0s 13us/step - loss: 0.0578 - val_loss: 0.0101
    Epoch 557/1500
    6484/6484 [==============================] - 0s 12us/step - loss: 0.0578 - val_loss: 0.0101
    Epoch 558/1500
    6484/6484 [==============================] - 0s 14us/step - loss: 0.0578 - val_loss: 0.0100
    Epoch 559/1500
    6484/6484 [==============================] - 0s 13us/step - loss: 0.0573 - val_loss: 0.0099
    Epoch 560/1500
    6484/6484 [==============================] - 0s 13us/step - loss: 0.0577 - val_loss: 0.0099
    Epoch 561/1500
    6484/6484 [==============================] - 0s 14us/step - loss: 0.0570 - val_loss: 0.0100
    Epoch 562/1500
    6484/6484 [==============================] - 0s 12us/step - loss: 0.0567 - val_loss: 0.0100
    Epoch 563/1500
    6484/6484 [==============================] - 0s 15us/step - loss: 0.0575 - val_loss: 0.0100
Epoch 00563: early stopping

The best iteration is here:

Epoch 473/1500
6484/6484 [==============================] - 0s 12us/step - loss: 0.0698 - val_loss: 0.0096

How can I extract this score 0.0096, for example as a scoring output for bayes optimization or SMAC? (i.e. given md, I need md.min_val_loss()) I tried:

    print(md.history.keys())
Traceback (most recent call last):

  File "<ipython-input-100-d1e5bda1287c>", line 1, in <module>
    print(md.history.keys())

AttributeError: 'History' object has no attribute 'keys'




print(md.history['val_loss'])
Traceback (most recent call last):

  File "<ipython-input-101-37ce8f0572c5>", line 1, in <module>
    print(md.history['val_loss'])

TypeError: 'History' object is not subscriptable

Apperently it doesn't work. BTW, can I do my prediction up to the best iteration? something like: md.predict(new_data, iteration=473)

score 4 · Accepted Answer · answered Sep 04 '18 at 06:09

Keras allows you to only store the results of the best model, which might be what you are looking for:
You can simply pass another callback to your .fit() call, something like this:

checkpoint = ModelCheckpoint("keras_model.pt", monitor='val_loss', save_best_only=True)
model.fit(...., callbacks=[...,checkpoint]) # attach callback to training!

That way, you can reload your model once training finished, and predict from there. As for the question on how to retrieve the value after, have a look at this post on SO. Namely, simply create your own History() (I had to chuckle after re-reading the sentence).

from keras.callbacks import History
history = History()
hist = model.fit(..., callbacks=[...,history])
print(hist.history)

To access loss for a specific epoch, do hist.history['val_loss'][<epoch>].

keras extract best val loss

1 Answers1