Keras: what does the (x, x) here mean? x = layers.MultiHeadAttention(key_dim=head_size, num_heads=num_heads, dropout=dropout)(x, x)

Question

In this example from Keras there is a code like this:

def transformer_encoder(inputs, head_size, num_heads, ff_dim, dropout=0):
    # Normalization and Attention
    x = layers.LayerNormalization(epsilon=1e-6)(inputs)
    x = layers.MultiHeadAttention(
        key_dim=head_size, num_heads=num_heads, dropout=dropout
    )(x, x)
    x = layers.Dropout(dropout)(x)
    res = x + inputs

What does the (x, x) mean on the sixth line?

Can I change the second x to be another input? (I am trying to implement this model here and I don't know how to add the additional data as query input, maybe changing the second x would be it?).

The `(x, x)` are the `query` and `value` call arguments that you can see described in [docs](https://keras.io/api/layers/attention_layers/multi_head_attention/). Call arguments are passed to the instance's `_call_` method (see [here](https://stackoverflow.com/questions/44178162/call-method-of-type-class) for a basic example). In short, yes, you can change the second `x`. — AlexK, Sep 05 '22 at 04:19

Keras: what does the (x, x) here mean? x = layers.MultiHeadAttention(key_dim=head_size, num_heads=num_heads, dropout=dropout)(x, x)

0 Answers0