Adding a Concatenated layer to TensorFlow 2.0 (using Attention)

Question

In building a model that uses TensorFlow 2.0 Attention I followed the example given in the TF docs. https://www.tensorflow.org/api_docs/python/tf/keras/layers/Attention

The last line in the example is

input_layer = tf.keras.layers.Concatenate()(
    [query_encoding, query_value_attention])

Then the example has the comment

# Add DNN layers, and create Model.
# ...

So it seemed logical to do this

model = tf.keras.Sequential()
model.add(input_layer)

This produces the error

TypeError: The added layer must be an instance of class Layer.
Found: Tensor("concatenate/Identity:0", shape=(None, 200), dtype=float32)

UPDATE (after @thushv89 response)

What I am trying to do in the end is add an attention layer in the following model which works well (or convert it to an attention model).

model = tf.keras.Sequential()
model.add(layers.Embedding(vocab_size, embedding_nodes, input_length=max_length))
model.add(layers.LSTM(20))
#add attention here?
model.add(layers.Dense(1, activation='sigmoid'))
model.compile(loss='mean_squared_error', metrics=['accuracy'])

My data looks like this

4912,5059,5079,0
4663,5145,5146,0
4663,5145,5146,0
4840,5117,5040,0

Where the first three columns are the inputs and the last column is binary and the goal is classification. The data was prepared similarly to this example with a similar purpose, binary classification. https://machinelearningmastery.com/use-word-embedding-layers-deep-learning-keras/

Answer 1

So, first thing is Keras has three APIs when it comes to creating models.

• Sequential - (Which is what you're doing here)
• Functional - (Which is what I'm using in the solution)
• Subclassing - Creating Python classes to represent custom models/layers

The way the model created in the tutorial is not to be used with sequential models but a model from the Functional API. So you got to do the following. Note that, I've taken the liberty of defining the dense layers with arbitrary parameters (e.g. number of output classes, which you can change as needed).

import tensorflow as tf

# Variable-length int sequences.
query_input = tf.keras.Input(shape=(None,), dtype='int32')
value_input = tf.keras.Input(shape=(None,), dtype='int32')

# ... the code in the middle

# Concatenate query and document encodings to produce a DNN input layer.
input_layer = tf.keras.layers.Concatenate()(
    [query_encoding, query_value_attention])

# Add DNN layers, and create Model.
# ...
dense_out = tf.keras.layers.Dense(50, activation='relu')(input_layer)
pred = tf.keras.layers.Dense(10, activation='softmax')(dense_out)

model = tf.keras.models.Model(inputs=[query_input, value_input], outputs=pred)
model.summary()