Topological sort failed when using Keras TimeDistributed layer

Question

I try to dot last column of lookback dimension to prior lookback period on 4D tensor (Sample, Time-steps, Lookback,features) by using keras TimeDistributed layer. The model can run normally but it throw a warning that graph couldn't be sorted in topological order when I run Model.fit().

This forum say that it can mess up model training. So what can I do to prevent this happen?

ENV:

1. Tensorflow-GPU 1.15.0
2. CUDA V10.0.130
3. python 3.6.5
4. Keras 2.3.1
5. Keras-Applications 1.0.8
6. Keras-Preprocessing 1.1.0

import numpy as np
from keras.models import Model
from keras.layers import Input, TimeDistributed
import keras
# Dot layer
class Dot(keras.layers.Layer):
    def __init__(self, **kwargs):
        super(Dot, self).__init__(**kwargs)

    def call(self, x):

        ht, hT = x[:,:-1,:],x[:,-1:,:]
        ml = tf.multiply(ht, hT)

        # I believe problem come from reduce_sum
        dot = tf.reduce_sum(ml, axis=-1)
        return dot

    def compute_output_shape(self, input_shape):

        return (None,input_shape[1]-1)

num_fea = 11
num_lookback = 5
time_step = 3
sample = 2

# create model
input = Input(shape=(time_step,num_lookback,num_fea))
dot = Dot()
output = TimeDistributed(dot)(input)

M = Model(inputs=[input], outputs=[output])
M.compile(keras.optimizers.Adam(learning_rate=0.0001), loss='mse')

# create test data
data = np.arange(num_lookback*num_fea).reshape((num_lookback,num_fea))
data = np.broadcast_to(data,shape=(sample,time_step,num_lookback,num_fea))
y = np.ones(shape=(sample,time_step,num_lookback-1))

# fit model to demonstrate error
M.fit(x=data,y=y, batch_size=2, epochs=10)

Warning log

2020-03-05 08:36:17.558396: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:697] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order.
2020-03-05 08:36:17.558777: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:533] layout failed: Invalid argument: The graph couldn't be sorted in topological order.
2020-03-05 08:36:17.559302: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:533] model_pruner failed: Invalid argument: MutableGraphView::MutableGraphView error: node 'loss/time_distributed_1_loss/mean_squared_error/weighted_loss/concat' has self cycle fanin 'loss/time_distributed_1_loss/mean_squared_error/weighted_loss/concat'.
2020-03-05 08:36:17.560121: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:533] remapper failed: Invalid argument: MutableGraphView::MutableGraphView error: node 'loss/time_distributed_1_loss/mean_squared_error/weighted_loss/concat' has self cycle fanin 'loss/time_distributed_1_loss/mean_squared_error/weighted_loss/concat'.
2020-03-05 08:36:17.560575: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:533] arithmetic_optimizer failed: Invalid argument: The graph couldn't be sorted in topological order.
2020-03-05 08:36:17.560853: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:697] Iteration = 0, topological sort failed with message: The graph couldn't be sorted in topological order.
2020-03-05 08:36:17.561141: E tensorflow/core/grappler/optimizers/dependency_optimizer.cc:697] Iteration = 1, topological sort failed with message: The graph couldn't be sorted in topological order.

Answer 1

you can consider using TensorFlow 2.x version.

I have migrated/upgraded your code and verify that it works on google colab. You can try to look here for more info on how to migrate your code to Tensorflow 2.x

Kindly refer to the code below

import numpy as np
import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, TimeDistributed
#import keras
# Dot layer
class Dot(tf.keras.layers.Layer):
    def __init__(self, **kwargs):
        super(Dot, self).__init__(**kwargs)

    def call(self, x):

        ht, hT = x[:,:-1,:],x[:,-1:,:]
        ml = tf.multiply(ht, hT)

        # I believe problem come from reduce_sum
        dot = tf.reduce_sum(ml, axis=-1)
        return dot

    def compute_output_shape(self, input_shape):

        return (None,input_shape[1]-1)

num_fea = 11
num_lookback = 5
time_step = 3
sample = 2

# create model
input = Input(shape=(time_step,num_lookback,num_fea))
dot = Dot()
output = TimeDistributed(dot)(input)

M = Model(inputs=[input], outputs=[output])
M.compile(optimizer='adam', loss='mse')

# create test data
data = np.arange(num_lookback*num_fea).reshape((num_lookback,num_fea))
data = np.broadcast_to(data,shape=(sample,time_step,num_lookback,num_fea))
y = np.ones(shape=(sample,time_step,num_lookback-1))

# fit model to demonstrate error
M.fit(x=data,y=y, batch_size=2, epochs=10)