0

I am trying to concatenate two models in keras but getting error

here are the two models

1. `image_model = Sequential([
    Dense(embedding_size, input_shape=(2048,), activation='relu'),
    RepeatVector(max_len)
])`
 2.` caption_model = Sequential([
    Embedding(vocab_size, embedding_size, input_length=max_len),
    LSTM(256, return_sequences=True),
    TimeDistributed(Dense(300))
])`

my concatenation function is 'concatenation' itself as we can't use merge in keras-2.0 so i have used it. 

   3. `final_model = Sequential([
    concatenate([image_model, caption_model],axis=-1),
    Bidirectional(LSTM(256, return_sequences=False)),
    Dense(vocab_size),
    Activation('softmax')
]) `

But here is the error i am getting how to resolve it i have googled it down but non of the solutions worked for me. please help

~/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py in 
assert_input_compatibility(self, inputs)
278             try:
279                 K.is_keras_tensor(x)
280             except ValueError:
~/anaconda3/lib/python3.6/site- 
packages/keras/backend/tensorflow_backend.py in is_keras_tensor(x)
471         raise ValueError('Unexpectedly found an instance 
ofstr(type(x)) + '`. '
473                          'Expected a symbolic tensor instance.')

ValueError: Unexpectedly found an instance of type `<class 
'keras.engine.sequential.Sequential'>`. Expected a symbolic tensor 
 instance.
 During handling of the above exception, another exception occurred:
 ValueError                                Traceback (most recent call 
  last)
 <ipython-input-107-04c9412f6b6d> in <module>
  5 
  6 final_model = Sequential([
  7         concatenate([image_model, caption_model],axis=-1),
  8         Bidirectional(LSTM(256, return_sequences=False)),
  9         Dense(vocab_size),

  ~/anaconda3/lib/python3.6/site-packages/keras/layers/merge.py in 
  concatenate(inputs, axis, **kwargs)
  639         A tensor, the concatenation of the inputs alongside axis 
  `axis`.
   640     """
   --> 641     return Concatenate(axis=axis, **kwargs)(inputs)
   642 
    643 

   ~/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py 
     in __call__(self, inputs, **kwargs)
    412                 # Raise exceptions in case the input is not 
    compatible
    413                 # with the input_spec specified in the layer 
    constructor.
    --> 414                 self.assert_input_compatibility(inputs)
    415 
    416                 # Collect input shapes to build layer.

    ~/anaconda3/lib/python3.6/site-packages/keras/engine/base_layer.py 
    in assert_input_compatibility(self, inputs)
    283                                  'Received type: ' +
   284                                  str(type(x)) + '. Full input: 
   ' +
  --> 285                                  str(inputs) + '. All inputs 
   to the layer '
286                                  'should be tensors.')
287 

   ValueError: Layer concatenate_16 was called with an input that 
  isn't a symbolic tensor. Received type: <class  
       keras.engine.sequential.Sequential'>. Full input: 
    [<keras.engine.sequential.Sequential object at 0x7f63ae5b8240>, 
    <keras.engine.sequential.Sequential object at 0x7f63ae592320>]. 
      All inputs to the layer should be tensors.
Ayush Kushwaha
  • 93
  • 1
  • 1
  • 9
  • Possible duplicate of https://stackoverflow.com/questions/46397258/how-to-merge-sequential-models-in-keras-2-0 – giser_yugang Mar 21 '19 at 03:01
  • @giser_yugang but still getting the error ValueError: Layer concatenate_21 was called with an input that isn't a symbolic tensor. Received type: . Full input: [, ]. All inputs to the layer should be tensors – Ayush Kushwaha Mar 21 '19 at 10:02
  • Keras `concatenate` layer does not support concate` Sequential model` type as stated in that answer. You should follow the answer and change your input to layer type . – giser_yugang Mar 22 '19 at 06:11
  • @giser_yugang thanks for your reply. please help it's been a while since i have stuck on this problem i wrote down all the codes after reading the article you have provided but still getting error.I am newbie in this thing can you please do convert those codes for me it would mean a lot to me . – Ayush Kushwaha Mar 26 '19 at 11:56
  • @giser_yugang i am working on image captioning problem on Flicker8k Dataset the train took encoding around 1 hrs and 45 minutes and test encoding took around 35 minutes after investing this long amount of time it's frustrating to start allover again so please help. – Ayush Kushwaha Mar 26 '19 at 12:04

1 Answers1

1

As I said keras concatenate does not support concate Sequential model type. You should change your final_model to Keras functional Model. As follows:

concat_layers = concatenate([image_model.output, caption_model.output])
layer = Bidirectional(LSTM(256, return_sequences=False))(concat_layers)
layer = Dense(vocab_size)(layer)
outlayer = Activation('softmax')(layer)
final_model = Model([image_model.input, caption_model.input], [outlayer])
giser_yugang
  • 6,058
  • 4
  • 21
  • 44