Tensorflow Graph - check if a node depends on a placeholder

Question

In a Tensorflow graph, is there a way to find out if a node depends on a placeholder, like node.depends(placeholder) -> Bool

import tensorflow as tf

x = tf.placeholder(name='X', dtype=tf.int64, shape=[])
y = tf.placeholder(name='Y', dtype=tf.int64, shape=[])

p = tf.add(x, y)
q = tf.add(x, x)
sess = tf.Session()

result = sess.run([p,q], feed_dict={x: 1, y: 2})
print(result)

result = sess.run([p,q], feed_dict={x: 1, y: 3})
print(result)

In the code example above, q does not depend on y. In the second call of session.run, we modify only y. Thus q does not need to be evaluated again. Does session automatically reuse existing values in these cases? If so, is there any way to find out which are the nodes that were evaluated during .run?

Otherwise, if I can quickly find out which nodes are dependent on the placeholders I modify, I can send only those as input to run, and reuse existing values (in a dictionary as cache).

The idea is to avoid costly evaluations and more importantly, in my application, to minimize costly operations (outside tensorflow) that need to be triggered whenever the output nodes change - a necessity in my application.

score 1 · Accepted Answer · answered Oct 22 '19 at 17:58

Checking dependency between tensors in a graph can be done with a function like this:

import tensorflow as tf

# Checks if tensor a depends on tensor b
def tensor_depends(a, b):
    if a.graph is not b.graph:
        return False
    gd = a.graph.as_graph_def()
    gd_sub = tf.graph_util.extract_sub_graph(gd, [a.op.name])
    return b.op.name in {n.name for n in gd_sub.node}

For example:

import tensorflow as tf

x = tf.placeholder(name='X', dtype=tf.int64, shape=[])
y = tf.placeholder(name='Y', dtype=tf.int64, shape=[])
p = tf.add(x, y)
q = tf.add(x, x)
print(tensor_depends(q, x))
# True
print(tensor_depends(q, y))
# False

About your questions, generally TensorFlow does recompute everything on each run, even if the inputs do not change. If you want to cache results, you can do it yourself, at a higher level - for TensorFlow, it would not be clear what results it should keep (only the latest ones, a few recent ones, ...). And in any case, even if the inputs do not change, the output may change, as is the case with recurrent models, or more generally due to any change in stateful objects such as variables or datasets. There are some optimization opportunities that are lost, but it would probably not be reasonable to expect TensorFlow to address them (analyze the model to determine whether it can cache results or not, what results can be cached, how much, how to configure it, etc.). If you know that some intermediate values in the graph will not change, you can also compute that part and then feed it as input (you can actually feed values for any tensor in the feed_dict). Otherwise, as you suggest, you can just check what depends on what and recompute only as necessary.

Tensorflow Graph - check if a node depends on a placeholder

1 Answers1