keras : add layers to another model

Question

I need to add layers to an existing model. However, I need to add the layers at "the main model level", that is I can't use the classic functional approach. For example, if I use something like:

from keras.layers import Dense,Reshape, Input
inp = Input(shape=(15,))
d1 = Dense(224*224*3, activation='linear')(inp)
r1 = Reshape(input_shape)
from keras import Model
model_mod = r1(d1)
model_mod = mobilenet(model_mod)
model_mod = Model(inp, model_mod)

I obtain:

Layer (type)                 Output Shape              Param #   
=================================================================
input_5 (InputLayer)         (None, 15)                0         
_________________________________________________________________
dense_4 (Dense)              (None, 150528)            2408448   
_________________________________________________________________
reshape_4 (Reshape)          (None, 224, 224, 3)       0         
_________________________________________________________________
mobilenet_1.00_224 (Model)   (None, 1000)              4253864 

So, I obtain a model with a nested submodel. Instead, I would the nested submodel's layers (mobilenet) "added" to the new top layers (that is, after reshape_4). I tried with:

modelB_input = modelB.input
for layer in modelB.layers:
    if layer == modelB_input:
        continue
    modelA.add(layer)  

It works for simple sequential models (e.g., vgg, mobilenet) but with more complex models with connections not strictly sequential (e.g., inception,resnet) this code is not good. Any ideas?

Answer 1

You can use keras.layers.Concatenate to merge two models like so:

first = Sequential()
first.add(Dense(1, input_shape=(2,), activation='sigmoid'))

second = Sequential()
second.add(Dense(1, input_shape=(1,), activation='sigmoid'))
 
merged = Concatenate([first, second])

(Taken from: How to concatenate two layers in keras?)

Although this example uses keras.models.Sequential, it works for other models or layers as well.

You can also take a look at: https://keras.io/api/layers/merging_layers/concatenate/

Answer 2

If you want to add a A layer to a B layer in the existed model, you can get the B layer output to the A layer and parse them to a new model by tf.keras.model.Model. An comprehensive demonstration for this method is in the feature extractor for object detection or segmentation. You can found one in here

For example by adding 2 new layers to VGG16 model at the bottom

full_vgg_model = tf.keras.applications.VGG16(
                            include_top=False,
                            weights="imagenet",
                            input_tensor=None,
                            input_shape=None,
                            pooling=None,
                            classes=1000,
                        )

The current layers:

Model: "vgg16"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, None, None, 3)]   0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, None, None, 64)    1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, None, None, 64)    36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, None, None, 64)    0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, None, None, 128)   73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, None, None, 128)   147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, None, None, 128)   0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, None, None, 256)   295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, None, None, 256)   590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, None, None, 256)   590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, None, None, 256)   0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, None, None, 512)   1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, None, None, 512)   0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, None, None, 512)   0         
=================================================================
Total params: 14,714,688
Trainable params: 14,714,688
Non-trainable params: 0

Then I append 2 new layers:

conv6 = tf.keras.layers.Conv2D(1024, 3, strides=(1, 1), padding='same', activation='relu', dilation_rate=(6,6), name='conv6')(full_vgg_model.layers[-1].output)

conv7 = tf.keras.layers.Conv2D(1024, 1, strides=(1, 1), padding='same', activation='relu', name='conv7')(conv6)
    
classification_backbone = tf.keras.Model(
            inputs=full_vgg_model.inputs,
            outputs=[conv6,conv7])

We got them stacked at the bottom!

Model: "model"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         [(None, None, None, 3)]   0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, None, None, 64)    1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, None, None, 64)    36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, None, None, 64)    0         
_________________________________________________________________
block2_conv1 (Conv2D)        (None, None, None, 128)   73856     
_________________________________________________________________
block2_conv2 (Conv2D)        (None, None, None, 128)   147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, None, None, 128)   0         
_________________________________________________________________
block3_conv1 (Conv2D)        (None, None, None, 256)   295168    
_________________________________________________________________
block3_conv2 (Conv2D)        (None, None, None, 256)   590080    
_________________________________________________________________
block3_conv3 (Conv2D)        (None, None, None, 256)   590080    
_________________________________________________________________
block3_pool (MaxPooling2D)   (None, None, None, 256)   0         
_________________________________________________________________
block4_conv1 (Conv2D)        (None, None, None, 512)   1180160   
_________________________________________________________________
block4_conv2 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block4_conv3 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block4_pool (MaxPooling2D)   (None, None, None, 512)   0         
_________________________________________________________________
block5_conv1 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_conv2 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_conv3 (Conv2D)        (None, None, None, 512)   2359808   
_________________________________________________________________
block5_pool (MaxPooling2D)   (None, None, None, 512)   0         
_________________________________________________________________
conv6 (Conv2D)               (None, None, None, 1024)  4719616   
_________________________________________________________________
conv7 (Conv2D)               (None, None, None, 1024)  1049600   
=================================================================
Total params: 20,483,904
Trainable params: 20,483,904
Non-trainable params: 0