8

I am trying to create my first ensemble models in keras. I have 3 input values and a single output value in my dataset.

from keras.optimizers import SGD,Adam
from keras.layers import Dense,Merge
from keras.models import Sequential

model1 = Sequential()
model1.add(Dense(3, input_dim=3, activation='relu'))
model1.add(Dense(2, activation='relu'))
model1.add(Dense(2, activation='tanh'))
model1.compile(loss='mse', optimizer='Adam', metrics=['accuracy'])

model2 = Sequential()
model2.add(Dense(3, input_dim=3, activation='linear'))
model2.add(Dense(4, activation='tanh'))
model2.add(Dense(3, activation='tanh'))
model2.compile(loss='mse', optimizer='SGD', metrics=['accuracy'])

model3 = Sequential()
model3.add(Merge([model1, model2], mode = 'concat'))
model3.add(Dense(1, activation='sigmoid'))
model3.compile(loss='binary_crossentropy', optimizer='Adam', metrics=['accuracy'])

model3.input_shape

The ensemble model(model3) compiles without any error but while fitting the model I have to pass the same input two times model3.fit([X,X],y). Which I think is an unnecessary step and instead of passing input twice I want to have a common input nodes for my ensemble model. How can I do it?

Eka
  • 14,170
  • 38
  • 128
  • 212

3 Answers3

7

Keras functional API seems to be a better fit for your use case, as it allows more flexibility in the computation graph. e.g.:

from keras.layers import concatenate
from keras.models import Model
from keras.layers import Input, Merge
from keras.layers.core import Dense
from keras.layers.merge import concatenate

# a single input layer
inputs = Input(shape=(3,))

# model 1
x1 = Dense(3, activation='relu')(inputs)
x1 = Dense(2, activation='relu')(x1)
x1 = Dense(2, activation='tanh')(x1)

# model 2 
x2 = Dense(3, activation='linear')(inputs)
x2 = Dense(4, activation='tanh')(x2)
x2 = Dense(3, activation='tanh')(x2)

# merging models
x3 = concatenate([x1, x2])

# output layer
predictions = Dense(1, activation='sigmoid')(x3)

# generate a model from the layers above
model = Model(inputs=inputs, outputs=predictions)
model.compile(optimizer='adam',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Always a good idea to verify it looks as you expect it to 
# model.summary()

data = [[1,2,3], [1,1,3], [7,8,9], [5,8,10]]
labels = [0,0,1,1]

# The resulting model can be fit with a single input:
model.fit(data, labels, epochs=50)

Notes:

  • There might be slight differences in the API between Keras versions (pre- and post- version 2)
  • The example above specifies different optimizer and loss function for each of the models. However, since fit() is being called only once (on model3), the same settings - those of model3 - will apply to the entire model. In order to have different settings when training the sub-models, they will have to be fit() separately - see comment by @Daniel.

EDIT: updated notes based on comments

etov
  • 2,972
  • 2
  • 22
  • 36
  • The compilation (optimizer and loss) of a model is only taken into account when you use **`fit` for that specific model**. If you are using **`fit` in `model3`**, only the compilation for `model3` will take effect. --- There is no need at all to compile `model1` and `model2`, unless you're going to train them separately (with `model1.fit` and `model2.fit`). Weights and predictions do not require `compile`. – Daniel Möller Sep 01 '17 at 16:27
6

etov's answer is a great option.

But suppose you already have model1 and model2 ready and you don't want to change them, you can create the third model like this:

singleInput = Input((3,))

out1 = model1(singleInput)   
out2 = model2(singleInput)
#....
#outN = modelN(singleInput)

out = Concatenate()([out1,out2]) #[out1,out2,...,outN]
out = Dense(1, activation='sigmoid')(out)

model3 = Model(singleInput,out)

And if you already have all the models ready and don't want to change them, you can have something like this (not tested):

singleInput = Input((3,))
output = model3([singleInput,singleInput])
singleModel = Model(singleInput,output)
Daniel Möller
  • 84,878
  • 18
  • 192
  • 214
  • This is indeed a good option, and in practice I think the two ways are nearly equivalent, with the exception of the input layer being an embedding. In that case using a common input layer vs. using a different one for each model would make a difference (both are valid - the right choice would depend on the application) – etov Sep 01 '17 at 09:18
  • Yes. But the resulting model is not sequential. – Daniel Möller Sep 26 '18 at 03:38
3

Define new input layer and use model outputs directly (works in functional api):

assert model1.input_shape == model2.input_shape # make sure they got same shape

inp = tf.keras.layers.Input(shape=model1.input_shape[1:])
model = tf.keras.models.Model(inputs=[inp], outputs=[model1(inp), model2(inp)])
Mendi Barel
  • 3,350
  • 1
  • 23
  • 24