Get the layer output of a submodel in Keras

Question

I am currently trying to implement a custom loss function in Keras. However, as my model is a combination of VGG and my custom network, I can't get the output of my submodel.

Here is the network definition:

model = VGG19(weights='imagenet', include_top=False)
#model.summary()

def change_model(model, new_input_shape=(None, 40, 40, 3)):
    ''' Change the input size of the provided network'''
    # replace input shape of first layer
    model._layers[0].batch_input_shape = new_input_shape

    # rebuild model architecture by exporting and importing via json
    new_model = keras.models.model_from_json(model.to_json())

    # copy weights from old model to new one
    for layer in new_model.layers:
        try:
            layer.set_weights(model.get_layer(name=layer.name).get_weights())
            print("Loaded layer {}".format(layer.name))
        except:
            print("Could not transfer weights for layer {}".format(layer.name))

    return new_model

new_model = change_model(model,new_input_shape=(None, 1024, 1024, 3))
new_model.summary()
for layer in new_model.layers:
    layer.trainable = False

vector_1 = new_model.get_layer("block4_conv4").output

def create_detector_network(kernel_reg = 0.):
    input = Input(shape=(128, 128, 512))
    x = Conv2D(128, kernel_size=3, strides=1, name='detect_1', padding='same', kernel_regularizer=regularizers.l2(kernel_reg))(input)
    x = BatchNormalization()(x)
    x = Conv2D(1+pow(8,2), kernel_size=1, strides=1, name='detect_2', kernel_regularizer=regularizers.l2(kernel_reg))(x)
    x = BatchNormalization()(x)
    prob = Activation('softmax')(x)
    prob = Lambda(lambda x: x[:,:, :, :-1], output_shape= (128, 128, 64))(prob)  #x[:, :, :-1]
    prob = keras.layers.UpSampling2D(size=(8, 8), data_format=None, interpolation='nearest')(prob)
    prob = Conv2D(1, kernel_size=1, strides=1, name='reduce_dim')(prob)

    return Model(input, [prob, x])



detector_model = create_detector_network()

detector_model.summary()

output = detector_model(vector_1)

full_model  = Model(inputs=new_model.input, outputs=output)

and the summary can be seen here: My Network:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_4 (InputLayer)         (None, 128, 128, 512)     0         
_________________________________________________________________
detect_1 (Conv2D)            (None, 128, 128, 128)     589952    
_________________________________________________________________
batch_normalization_3 (Batch (None, 128, 128, 128)     512       
_________________________________________________________________
detect_2 (Conv2D)            (None, 128, 128, 65)      8385      
_________________________________________________________________
batch_normalization_4 (Batch (None, 128, 128, 65)      260       
_________________________________________________________________
activation_2 (Activation)    (None, 128, 128, 65)      0         
_________________________________________________________________
lambda_2 (Lambda)            (None, 128, 128, 64)      0         
_________________________________________________________________
up_sampling2d_2 (UpSampling2 (None, 1024, 1024, 64)    0         
_________________________________________________________________
reduce_dim (Conv2D)          (None, 1024, 1024, 1)     65        
=================================================================
Total params: 599,174
Trainable params: 598,788
Non-trainable params: 386
_________________________________________________________________

VGG + My Network:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_7 (InputLayer)         (None, 1024, 1024, 3)     0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 1024, 1024, 64)    1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 1024, 1024, 64)    36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 512, 512, 64)      0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, 512, 512, 128)     73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, 512, 512, 128)     147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 256, 256, 128)     0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, 256, 256, 256)     295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, 256, 256, 256)     590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, 256, 256, 256)     590080    
_________________________________________________________________
block3_conv4 (Conv2D)        (None, 256, 256, 256)     590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, 128, 128, 256)     0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, 128, 128, 512)     1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, 128, 128, 512)     2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, 128, 128, 512)     2359808   
_________________________________________________________________
block4_conv4 (Conv2D)        (None, 128, 128, 512)     2359808   
_________________________________________________________________
model_7 (Model)              (None, 128, 128, 65)      599109    
=================================================================
Total params: 11,184,261
Trainable params: 598,723
Non-trainable params: 10,585,538
_________________________________________________________________

This is how I train my network:

losses = {
    "detect_2": "mse",
    "reduce_dim": "mse",
}

full_model.compile(optimizer='adam',
              loss=losses,
              loss_weights={'prob': 1.0, 'main': 0})

full_model.summary()

history = full_model.fit_generator(
    train_generator,
    steps_per_epoch=100,
    epochs=7,
    validation_data=validation_generator,
    validation_steps=80)

Now, I want to use the output of the layer 'detect_2' and 'reduce_dim' and compute the loss/accuracy from these. But, when I run my code I get the following error:

ValueError: Unknown entry in loss dictionary: "detect_2". Only expected the following keys: ['model_7', 'model_7']

Obviously, there must be an error somewhere, as a dictionary cannot have the same key twice. So could anybody tell me what I need to change such that I get the output of the layers?

score 0 · Answer 1 · answered May 02 '20 at 18:10

If 'detect_2' and 'reduce_dim' are layers of your model you need to specify their outputs as an output of the network. Then, you can access them in a custom loss via 'y_true' and 'y_pred' which will hold all network outputs for the ground truth and prediction respectively. Something like:

 return Model([input], [detect_2_output, reduce_dim_output])

Then in a custom loss use them however you like

def custom_loss(y_true, y_pred):   
    return

Get the layer output of a submodel in Keras

1 Answers1