Ensemble forecast with keras on GPU

Question

I am trying to make an ensemble forecast with equally build keras models. The Data for the single NNs are of the same shape. I want to use the GPU because the models should be trained parallel. Therefor I am trying to merge the models. Because the number of models should be configurable, I want to do this in a loop. I found some solutions, but for each I have troubles with the loop. Here is my approach:

from keras import Sequential, Model
from keras.layers import Embedding, GlobalAveragePooling1D, Dense, concatenate
import numpy as np

nummodels=3

model = Sequential()
model.add(Embedding(20, 10, trainable=True))
model.add(GlobalAveragePooling1D())
model.add(Dense(1, activation='sigmoid'))

for i in range(nummodels-1):
    model2 = Sequential()
    model2.add(Embedding(20, 10, trainable=True))
    model2.add(GlobalAveragePooling1D())
    model2.add(Dense(1, activation='sigmoid'))
    
    model_concat = concatenate([model.output, model2.output], axis=-1)
    model_concat = Dense(1, activation='softmax')(model_concat)
    model = Model(inputs=[model.input, model2.input], outputs=model_concat)

model.compile(loss='binary_crossentropy', optimizer='adam')

# somehow generating testdata x1,x2,x3 and y1,y2,y3...
# not implemented yet...

# Training
model.fit([x1,x2,x3],[y1,y2,y3], epochs = 50)
# prediction
ypred1,ypred2,ypred3 = model.predict([x1,x2,x3])

The loop is not working. I hope you can tell me what's the problem. Later on I will train and predict also in a loop. Is the fitting and prediction correct in my code? If it's not possible to do this in a loop, please give me other solutions.

EDIT:

Based on the Answer from M.Innat, I changed my code. Now, in the loop I add the inputs and outputs of the NNs to a list which I use lateron for concaternation. But the concaternation after the first loop is still not working.

from keras import Model
from keras.layers import Dense, Input
import numpy as np
import tensorflow as tf

nummodels=3

inputs=[]
outputs=[]
final_outputs=[]
init = 'uniform'
activation = 'tanh'
for i in range(nummodels):
    
    input_layer = Input(shape=(11,))
    A2 = Dense(8, kernel_initializer=init, activation=activation)(input_layer)
    A2 = Dense(5, kernel_initializer=init, activation=activation)(A2)
    A2 = Dense(3, kernel_initializer=init, activation=activation)(A2)
    A2 = Dense(1, kernel_initializer=init, activation=activation)(A2)
    model = Model(inputs=input_layer, outputs=A2, name="Model"+str(i+1))
       
    inputs.append(model.input)
    outputs.append(model.output)
    
model_concat = tf.keras.layers.concatenate(outputs, name='target_concatenate')    

for i in range(nummodels):
    final_outputs.append(Dense(1, activation='sigmoid')(model_concat))
   
# whole models 
composed_model = tf.keras.Model(inputs=inputs, outputs=final_outputs)

# model viz
tf.keras.utils.plot_model(composed_model)

# compile the model
composed_model.compile(loss='binary_crossentropy', optimizer='adam')

# data generation 
x1 = np.random.randint(1000, size=(32, 11))
x2 = np.random.randint(1000, size=(32, 11))
x3 = np.random.randint(1000, size=(32, 11))
y1 = np.random.randint(1000, size=(32, 1))
y2 = np.random.randint(1000, size=(32, 1))
y3 = np.random.randint(1000, size=(32, 1))

# Training
composed_model.fit([x1,x2,x3],[y1,y2,y3], epochs = 50)
# prediction
y1p, y2p, y3p = composed_model.predict([x1,x2,x3])

Answer 1

Firstly, you are trying to build nummodels times models and concate their output. And secondly, you try to .fit the model with multi-input and try to get multi-output - that means, your every model would take separately inputs and the merged model would give nummodels times output. So, based on my understanding of your question here is one possible solution for you.

---

Models

import tensorflow as tf
from tensorflow.keras import Input, Model
from tensorflow.keras.layers import Dense, Embedding, GlobalAveragePooling1D

# Define Model A 
input_layer = Input(shape=(10,))
A2 = Embedding(1000, 64)(input_layer)
A2 = GlobalAveragePooling1D()(A2)
model_a = Model(inputs=input_layer, outputs=A2, name="ModelA")

# Define Model B 
input_layer = Input(shape=(10,))
A2 = Embedding(1000, 64)(input_layer)
A2 = GlobalAveragePooling1D()(A2)
model_b = Model(inputs=input_layer, outputs=A2, name="ModelB")

# Define Model C
input_layer = Input(shape=(10,))
A2 = Embedding(1000, 64)(input_layer)
A2 = GlobalAveragePooling1D()(A2)
model_c = Model(inputs=input_layer, outputs=A2, name="ModelC")

Merge Model

import numpy as np 
import tensorflow as tf 

# concate their output 
model_concat = tf.keras.layers.concatenate(
    [
     model_a.output, model_b.output, model_c.output
    ], name='target_concatenate')

# final activation layer 
final_out1 = Dense(1, activation='sigmoid')(model_concat)
final_out2 = Dense(1, activation='sigmoid')(model_concat)
final_out3 = Dense(1, activation='sigmoid')(model_concat)

# whole models 
composed_model = tf.keras.Model(
    inputs=[model_a.input, model_b.input, model_c.input],
    outputs=[final_out1, final_out2, final_out3]
)

# model viz
tf.keras.utils.plot_model(composed_model)

I/O

# compile the model
composed_model.compile(loss='binary_crossentropy', optimizer='adam')

# dummies 
input_array1 = np.random.randint(1000, size=(32, 10))
input_array2 = np.random.randint(1000, size=(32, 10))
input_array3 = np.random.randint(1000, size=(32, 10))

# get some prediction - multi-input / multi-output
output_array1, output_array2, output_array3 = composed_model.predict([input_array1,
                                                                      input_array2,
                                                                      input_array3])

---

Check these answers for more understanding.

• Multi-input Multi-output Model with Keras Functional API
• Merge 3 Deep Network and Train End-to-End
• Training Multiple Input and Output Keras Model

---

Based on your comment and edited part of your question, try as follows:

models = []
for i in range(nummodels):
    ....
    ....
    model = Model(inputs=input_layer, outputs=A2, name="Model"+str(i+1))

    models.append(model)
    # inputs.append(model.input)
    # outputs.append(model.output)

# model_concat = tf.keras.layers.concatenate(outputs, name='target_concatenate')    
model_concat = tf.keras.layers.concatenate(
    [
     models[0].output, models[1].output, models[2].output
    ], name='target_concatenate')

final_outputs=[]
for i in range(nummodels):
    final_outputs.append(Dense(1, activation='sigmoid')(model_concat))
   
# whole models 
composed_model = tf.keras.Model(inputs=[models[0].input, 
                                        models[1].input, 
                                        models[2].input], 
                                outputs=final_outputs)