Output shape of lambda layer not right in Neural Net. How change it?

Question

this is my first question on Stackoverflow, so if I missed somehting please point it out to me. I have a Problem with my Lambda layer using keras and tensorflow 1. In this Lambda layer I am taking a 100-dimensional glove Vector as Input and compute cosine similarity to 8 other vectors (I converted to Tensors previously). As ouput I want the eight resulting cosine similarities as a Tensor (I thought this is necessary in tensorflow?).

My Problem now is that the shape of the resulting Tensor obviously is (8, 1), but actually I think I Need the Output shape (None, 8). Otherwise it will not match the subsequent layer in my Network which is the Output layer and should Output six class probabilities.

This is the Code for my custom function I feed into the Lambda layer and took from Sentence similarity using keras:

from keras import backend as K

def cosine_distance(ref_vector):
    sess = K.get_session()
    global emo_vec_array
    ref_vector = K.l2_normalize(ref_vector, axis=-1)

    cos_sim_list = []
    for emo_vector in emo_vec_array:
      emo_vector = K.l2_normalize(emo_vector, axis=-1)
      cos_sim = K.mean(ref_vector * emo_vector, axis=-1, keepdims=True)*100
      cos_sim_list.append(cos_sim[0])
    return tf.convert_to_tensor(cos_sim_list)

def cos_dist_output_shape(shapes):
    shape1, shape2 = shapes
    return (shape1, 8)

test_vector = tf.convert_to_tensor(embeddings_index['happy'], dtype='float32')

test_result = cosine_distance(test_vector)
array = sess.run(test_result)

Output here, when printing the test result and the converted Tensor is:

Tensor("packed_53:0", shape=(8,), dtype=float32)
[0.5166239  0.2958691  0.317714   0.44583628 0.39608976 0.4195615 0.6432581  0.2618766 ]

The result is as I want it, but the Output shapes in my NN are not Right. These are the last few layers with the respective Output shapes following:

hidden = Dense(vector_dimension, activation='relu')(attention)

distance = Lambda(cosine_distance)(hidden)

out = Dense(6, activation='softmax')(distance)


dense_41 (Dense)             (None, 100)               20100     
_________________________________________________________________
lambda_26 (Lambda)           (8, 1)                    0         
_________________________________________________________________
dense_42 (Dense)             (8, 6)                    12

What I want at the end is the following:

dense_41 (Dense)             (None, 100)               20100     
_________________________________________________________________
lambda_26 (Lambda)           (None, 8)                    0         
_________________________________________________________________
dense_42 (Dense)             (None, 6)                    12

I already tried K.transpose-ing the Tensor and experimenting with the Output-shape-function but that hadn't the desired effect. Any help would be very highly appreciated.

I hope I could make clear my Problem and thank you very much in Advance.

score 0 · Accepted Answer · answered Apr 02 '20 at 00:00

Simply change your cosine computation to a vectorized operation,

def cosine_dist(inp):
  # I decided to have this as a variable within the function. 
  # But you can also define this outside and pass it as an input to the function.
  emo_vectors = tf.ones(shape=(8,100))
  def normalize(x):
    return x / K.sum(x**2, axis=1, keepdims=True)

  inp = normalize(inp)
  emo_vectors = normalize(emo_vectors)  
  cdist = K.dot(inp, K.transpose(emo_vectors))

  return cdist

Here's an example of this in use,

inp = layers.Input(shape=(100))
hidden = layers.Lambda(lambda x: cosine_dist(x))(inp)
model = models.Model(inputs=inp, outputs=hidden)
model.summary()

Which gives,

Model: "model_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_8 (InputLayer)         [(None, 100)]             0         
_________________________________________________________________
lambda_7 (Lambda)            (None, 8)                 0         
=================================================================
Total params: 0
Trainable params: 0
Non-trainable params: 0
_________________________________________________________________

As you can see, the output of the lambda layer is (None, 8) now.

score 0 · Answer 2 · answered Apr 02 '20 at 07:39

After another while I was also able to find a second solution. The trick was to account for a flexible batch size. Here is the altered code of the cosine function:

from keras import backend as K

def cosine_distance(ref_vector):
    global emo_vec_array
    ref_vector = K.l2_normalize(ref_vector, axis=-1)

    cos_sim_list = []
    for emo_vector in emo_vec_array:
      emo_vector = K.l2_normalize(emo_vector, axis=-1)
      emo_vector = tf.reshape(emo_vector, [emo_vector.shape[0], 1])
      cos_sim = K.dot(ref_vector, emo_vector)
      cos_sim_list.append(cos_sim)

    result = tf.convert_to_tensor(cos_sim_list)
    result = tf.reshape(result, [len(emo_vec_array), -1])
    result = tf.transpose(result)
    return result

Output shape of lambda layer not right in Neural Net. How change it?

2 Answers2