0

I followed the official tutorial of the tensorflow website: https://www.tensorflow.org/extend/adding_an_op There is also described how to call the gradient of the example ZeroOut in the tutorial that I want to try in this short code snippet underneath.

I have found the code here: https://github.com/MatteoRagni/tf.ZeroOut.gpu

import numpy as np
import tensorflow as tf
from tensorflow.python.framework import ops
from tensorflow.python.ops import array_ops
from tensorflow.python.ops import sparse_ops
zero_out_module = tf.load_op_library('./libzeroout.so')
@ops.RegisterGradient("ZeroOut")
def _zero_out_grad(op, grad):
     to_zero = op.inputs[0]
     shape = array_ops.shape(to_zero)
     index = array_ops.zeros_like(shape)
     first_grad = array_ops.reshape(grad, [-1])[0]
     to_zero_grad = sparse_ops.sparse_to_dense([index], shape, first_grad, 0)
     return [to_zero_grad]  # List of one Tensor, since we have one input

t_in = tf.placeholder(tf.int32, [None,None])
ret = zero_out_module.zero_out(t_in)
grad = tf.gradients(ys=tf.reduce_sum(ret), xs=t_in)
with tf.Session(''):
    feed_dict = {t_in: [[1, 2], [3, 4]]}
    print "ret val: ", ret.eval(feed_dict=feed_dict)
    print "grad: ", grad
    print "grad: ", grad.eval(feed_dict=feed_dict)

I got this error ...

AttributeError: 'list' object has no attribute 'eval'

... but I can do ret.eval().

Why I cant call grad.eval()? I want to see these values inside the grad tensor. How to debug gradient?

j35t3r
  • 1,254
  • 2
  • 19
  • 53

1 Answers1

0

Answer to old question

The implementation

def _zero_out_grad(op, *grads):
    topdiff = grads[0]
    bottom = op.inputs[0]

    shape = array_ops.shape(bottom)
    index = array_ops.zeros_like(shape)
    first_grad = array_ops.reshape(topdiff, [-1])[0]
    to_zero_grad = sparse_ops.sparse_to_dense([index], shape, first_grad, 0)

    return to_zero_grad

works quite nicely here. Are you sure "@ops.RegisterGradient("ZeroOut")" is executed before the tf.Session()?

Usually the 

zero_out_module = tf.load_op_library('./libzeroout.so')
@ops.RegisterGradient("ZeroOut")
def _zero_out_grad(op, grad):
    # ...

is placed in a different file and just imported. A full working example even with the recent TensorFlow version is here.

Answer to completely changed question

Your gradient function returns a list and a Python list has no 'eval()'. Try either:

grad = tf.gradients(ys=tf.reduce_sum(ret), xs=t_in)[0]

Or follow best practice and use

grad = tf.gradients(ys=tf.reduce_sum(ret), xs=t_in)
with tf.Session() as sess:
    sess.run(grad, feed_dict=feed_dict)

Please do not change your entire question

Patwie
  • 4,360
  • 1
  • 21
  • 41