1

I have six different Sequnstial Keras models and I desire to concatenate them like the following:

Models

activation = 'tanh'
model1 = Sequential()
model1.add(Dense(3, input_dim=3, activation=activation))
model1.add(Dense(3, activation=activation))
model1.add(Dense(1))

model2 = Sequential()
model2.add(Dense(3, input_dim=3, activation=activation))
model2.add(Dense(3, activation=activation))
model2.add(Dense(1))

model3 = Sequential()
model3.add(Dense(1, input_dim=1, activation=activation))
model3.add(Dense(1, activation=activation))
model3.add(Dense(1))

model4 = Sequential()
model4.add(Dense(6, input_dim=6, activation=activation))
model4.add(Dense(3, activation=activation))
model4.add(Dense(1))

model5 = Sequential()
model5.add(Dense(2, input_dim=2, activation=activation))
model5.add(Dense(2, activation=activation))
model5.add(Dense(1))

model6 = Sequential()
model6.add(Dense(6, input_dim=6, activation=activation))
model6.add(Dense(4, activation=activation))
model6.add(Dense(1))

Concatenation

model_concat = concatenate([model1.output, model2.output, model3.output, model4.output, model5.output, model6.output])

composed_model = Model(
    inputs=[model1.input, model2.input, model3.input, model4.input, model5.input, model6.input],
    outputs=model_concat
)
composed_model.compile(optimizer=SGD(lr=0.001), loss='mse')

The output dimensions of all models are 1, but the input dimensions of models are different. The above code will run completely without any error.

Now, the problem is for training the model. I will try to run the following code to train the model with one data instance (the dimension of each subarray is equal to the input dimension of the corresponding model):

x_input = [[[-50.0, -50.0, 10.0], [-10.0, -20.0, 30.0], \
            [20.0], [10.0, 10.0, 10.0, 10.0, 10.0, 10.0], [50.0, 50.0],\
            [10.0, 10.0, 10.0, 10.0, 10.0, 10.0]]]

y_output = [[1.0, 1.0, 1.0, 1.0,1.0,1.0]]
composed_model.fit(x_input, y_output)

However, I have the following error:

ValueError: Failed to convert a NumPy array to a Tensor (Unsupported object type list).

I found a related post, but it does not solve my problem.

Innat
  • 16,113
  • 6
  • 53
  • 101
OmG
  • 18,337
  • 10
  • 57
  • 90

1 Answers1

1

As far as I can tell we can't just pass all variable-size inputs together in fit for the multi-input model. The way you pass your training pairs to the model, it surely unable to unpack for concern input layers. The related post that you mentioned is also an important fact to consider.

However, in tensorflow, we can use tf.ragged.RaggedTensor for variable-length input sequence, discussed here. But not sure if there is any workaround possible by converting to the ragged tensor. It probably would possible if a single input layer takes a different length of the input sequence.

If you read the fit method's training pairs input you would see keras expects the x paramer as follows:

Arguments

x: Input data. It could be:
    1. A Numpy array (or array-like), or a list of arrays (in case the model 
       has multiple inputs).
    2. A TensorFlow tensor, or a list of tensors (in case the model 
       has multiple inputs).
    3. A dict mapping input names to the corresponding array/tensors, 
       if the model has named inputs.
...
...

For your case, option 3 is pretty convenient to choose from, which is passing the dictionary mapped input names with training pairs. Here is one way we can do this. First set some names to the input layer of each model. We set model1, model2, ... etc.

from tensorflow.keras import Sequential
from tensorflow.keras.layers import Dense 

activation = 'tanh'
model1 = Sequential()
model1.add(Dense(3, input_dim=3, activation=activation, name='model1'))
model1.add(Dense(3, activation=activation))
model1.add(Dense(1))

model2 = Sequential()
model2.add(Dense(3, input_dim=3, activation=activation, name='model2'))
model2.add(Dense(3, activation=activation))
model2.add(Dense(1))

model3 = Sequential()
model3.add(Dense(1, input_dim=1, activation=activation, name='model3'))
model3.add(Dense(1, activation=activation))
model3.add(Dense(1))

model4 = Sequential()
model4.add(Dense(6, input_dim=6, activation=activation, name='model4'))
model4.add(Dense(3, activation=activation))
model4.add(Dense(1))

model5 = Sequential()
model5.add(Dense(2, input_dim=2, activation=activation, name='model5'))
model5.add(Dense(2, activation=activation))
model5.add(Dense(1))

model6 = Sequential()
model6.add(Dense(6, input_dim=6, activation=activation, name='model6'))
model6.add(Dense(4, activation=activation))
model6.add(Dense(1))

Now, build whole the final model where we also set the last layer name, which is set here as target_concatenate.

import tensorflow as tf 

model_concat = tf.keras.layers.concatenate(
    [
     model1.output, model2.output, model3.output, 
     model4.output, model5.output, model6.output
    ], name='target_concatenate')

composed_model = tf.keras.Model(
    inputs=[model1.input, model2.input, model3.input, 
            model4.input, model5.input, model6.input],
    outputs=model_concat
)
composed_model.compile(optimizer='sgd', loss='mse')

DataSet

The sample data you provided is not legal for model training as we mentioned above. Firstly it should not be a list but numpy and secondly, for multi-input variable size, it convenient to pass them separately.

import numpy as np 

# Six Inputs 
one = np.random.randint(-50, 10, size=(1, 3))
two = np.random.randint(-10, 30, size=(1, 3))
thr = np.random.randint(20, size=(1))
fur = np.random.randint(10, size=(1, 6))
fiv = np.random.randint(50, size=(1,2))
six = np.random.randint(10, size=(1, 6))

# One Target
tar = np.random.randint(2, size=(1, 6))

print(one, one.shape)
print(two, two.shape)
print(thr, thr.shape)
print(fur, fur.shape)
print(fiv, fiv.shape)
print(six, six.shape)
print(tar, tar.shape)

[[-42   9 -34]] (1, 3)
[[28 22  7]] (1, 3)
[19] (1,)
[[4 4 1 7 4 7]] (1, 6)
[[40 35]] (1, 2)
[[3 6 1 1 3 8]] (1, 6)
[[0 0 1 1 0 0]] (1, 6)

When we call .fit, we will pass these datasets as dict mapping model's input and output names to the corresponding array. So, let's check to get the names of the composed model.

# viewing the model shapes and layers name 
# tf.keras.utils.plot_model(composed_model, 
#                            show_shapes=True, show_layer_names=True)
composed_model.input_names, composed_model.output_names

(['model1_input',
  'model2_input',
  'model3_input',
  'model4_input',
  'model5_input',
  'model6_input'],
 ['target_concatenate'])

Great, now we can pass training Paris as follow conveniently to the fit method.

composed_model.fit(
    {
         "model1_input": one, "model2_input": two, "model3_input": thr,
         "model4_input": fur, "model5_input": fiv, "model6_input": six
     },
     {
         "target_concatenate": tar
     },
    epochs=10,
    batch_size=32,
    verbose=2
)

Epoch 1/10
496ms/step - loss: 0.6022
Epoch 2/10
5ms/step - loss: 0.5428
Epoch 3/10
7ms/step - loss: 0.4888
Epoch 4/10
16ms/step - loss: 0.4426
Epoch 5/10
8ms/step - loss: 0.4041
Epoch 6/10
8ms/step - loss: 0.3719
Epoch 7/10
9ms/step - loss: 0.3448
Epoch 8/10
5ms/step - loss: 0.3215
Epoch 9/10
7ms/step - loss: 0.3011
Epoch 10/10
5ms/step - loss: 0.2830
Innat
  • 16,113
  • 6
  • 53
  • 101