Keras - How to remove useless dimension without hurting the computation graph?

Question

While generating a deep learning model, I used K.squeeze function to squeeze useless dimension when the first two dimensions were None shape.

import keras.backend as K
>>> K.int_shape(user_input_for_TD)
(None, None, 1, 32)
>>> K.int_shape(K.squeeze(user_input_for_TD, axis=-2))
(None, None, 32)

However, this gives below error, It seems like K.squeeze function hurts the computation graph, is there any solution to escape from this issue? Maybe that function does not support calculating gradients, which isn't differentiable.

File "/home/sundong/anaconda3/envs/py36/lib/python3.6/site-packages/keras/engine/network.py", line 1325, in build_map
    node = layer._inbound_nodes[node_index]
AttributeError: 'NoneType' object has no attribute '_inbound_nodes'

Below code block is the whole code block which causes that error.

user_embedding_layer = Embedding(
            input_dim=len(self.data.visit_embedding),
            output_dim=32,
            weights=[np.array(list(self.data.visit_embedding.values()))],
            input_length=1,
            trainable=False)
...
all_areas_lstm = LSTM(1024, return_sequences=True)(all_areas_rslt)   # (None, None, 1024)
user_input_for_TD = Lambda(lambda x: x[:, :, 0:1])(multiple_inputs)  # (None, None, 1) 
user_input_for_TD = TimeDistributed(user_embedding_layer)(user_input_for_TD) # (None, None, 1, 32) 
user_input_for_TD = K.squeeze(user_input_for_TD, axis=-2) # (None, None, 32) 
aggre_threeway_inputs = Concatenate()([user_input_for_TD, all_areas_lstm]) # should be (None, None, 1056) 
threeway_encoder = TimeDistributed(ThreeWay(output_dim=512))
three_way_rslt = threeway_encoder(aggre_threeway_inputs) # should be (None, None, 512) 
logits = Dense(365, activation='softmax')(three_way_rslt) # should be (None, None, 365)
self.model = keras.Model(inputs=multiple_inputs, outputs=logits)

By removing below two lines (by not making it go through the embedding layer) , the code works without any issues. In this case, dimension of aggre_threeway_inputs = Concatenate()([user_input_for_TD, all_areas_lstm]) is (None, None, 1025).

user_input_for_TD = TimeDistributed(user_embedding_layer)(user_input_for_TD)
user_input_for_TD = K.squeeze(user_input_for_TD, axis=-2)

@HaBom That's a good pointer, is there any equivalent function in Keras? — SUNDONG, Mar 28 '19 at 09:34
you need to put your squeeze function inside a Lambda layer, you can't put tf operations directly into outputs of Keras layers — Dr. Snoopy, Mar 28 '19 at 11:23

SUNDONG · Accepted Answer · 2019-03-29T08:02:55.150

2

I solved it by using the Lambda layer with indexing, instead of K.squeeze function.

from keras.layers import Lambda
>>> K.int_shape(user_input_for_TD)
(None, None, 1, 32)
>>> K.int_shape(Lambda(lambda x: x[:, :, 0, :])(user_input_for_TD))
(None, None, 32)

edited Mar 29 '19 at 08:02

answered Mar 28 '19 at 09:41

SUNDONG

2,501
5
21
37

Keras - How to remove useless dimension without hurting the computation graph?

1 Answers1

Linked