Keras, get output of a layer at each epochs

Question

What I have done?

I implemented a keras model as follow:

train_X, test_X, train_Y, test_Y = train_test_split(X, Y, test_size=0.2, random_state=np.random.seed(7), shuffle=True)

train_X = np.reshape(train_X, (train_X.shape[0], 1, train_X.shape[1]))
test_X = np.reshape(test_X, (test_X.shape[0], 1, test_X.shape[1]))

model = Sequential()
model.add(LSTM(100, return_sequences=False, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(train_Y.shape[1], activation='softmax'))
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])

model.fit(train_X, train_Y, validation_split=.20,
                        epochs=1000, batch_size=50)

What i want?

I want to give support vector machine(SVM) the output of the penultimate layer (LSTM), in any epoch(that is 1000) to svm also be trained.

But I do not know how to do this?

Any idea?

UPDATED:

I use from ModelCheckpoint as follow:

model = Sequential()
model.add(LSTM(100, return_sequences=False, input_shape=(train_X.shape[1], train_X.shape[2])))
model.add(Dense(train_Y.shape[1], activation='softmax'))
model.compile(loss='mean_squared_error', optimizer='adam', metrics=['accuracy'])

# checkpoint
filepath="weights-{epoch:02d}-{val_acc:.2f}.hdf5"
checkpoint = ModelCheckpoint(filepath, monitor='val_acc', verbose=1, save_best_only=True, mode='max')
callbacks_list = [checkpoint]

model.fit(train_X, train_Y, validation_split=.20,
                    epochs=1000, batch_size=50, callbacks=callbacks_list, verbose=0)

OUTPUT:

Epoch 00991: val_acc did not improve
Epoch 00992: val_acc improved from 0.93465 to 0.93900, saving model to weights-992-0.94.hdf5
Epoch 00993: val_acc did not improve
Epoch 00994: val_acc did not improve
Epoch 00995: val_acc did not improve
Epoch 00996: val_acc did not improve
Epoch 00997: val_acc did not improve
Epoch 00998: val_acc improved from 0.93900 to 0.94543, saving model to weights-998-0.94.hdf5
Epoch 00999: val_acc did not improve

PROBLEM:

How to load all these models to obtain the output of the LSTM layer in each epochs as @IonicSolutions said?

Answer 1

What works best in your situation depends on how exactly you set up and train your SVM, but there are at least two options using callbacks:

You could use the ModelCheckpoint callback to save a copy of the model you are training at each epoch and then later load all these models to obtain the output of the LSTM layer.

You can also create your own callback by implementing the Callback base class. Within the callback, the model can be accessed and you can use on_epoch_end to extract the LSTM output at the end of each epoch.

Edit: To get convenient access to the penultimate layer, you can do the following:

# Create the model with the functional API
inp = Input((train_X.shape[1], train_X.shape[2],))
lstm = LSTM(100, return_sequences=False)(inp)
dense = Dense(train_Y.shape[1], activation='softmax')(lstm)

# Create the full model
model = Model(inputs=inp, outputs=dense)

# Create the model for access to the LSTM layer
access = Model(inputs=inp, outputs=lstm)

Then, you can pass access to your callback when you instantiate it. The key thing to note here is that model and access share the very same LSTM layer, whose weights will change when training model.

Answer 2

In order to get prediction output at each epoch here is what we can do:

import tensorflow as tf
import keras

# define your custom callback for prediction
class PredictionCallback(tf.keras.callbacks.Callback):    
  def on_epoch_end(self, epoch, logs={}):
    y_pred = self.model.predict(self.validation_data[0])
    print('prediction: {} at epoch: {}'.format(y_pred, epoch))

# ...

# register the callback before training starts
model.fit(X_train, y_train, batch_size=32, epochs=25, 
          validation_data=(X_valid, y_valid), 
          callbacks=[PredictionCallback()])