ValueError: Num gradients 1 generated for op name: "mask/Mask"

Question

I followed this link to create a customized op called mask. The main body of the tensorflow op is

def tf_mask(x, labels, epoch_, name=None):  # add "labels" to the input
    with ops.name_scope(name, "Mask", [x, labels, epoch_]) as name:
        z = py_func(np_mask,
                [x, labels, epoch_],   # add "labels, epoch_" to the input list
                [tf.float32],
                name=name,
                grad=our_grad)
        z = z[0]
        z.set_shape(x.get_shape())
        return z

which actually pretty much follows the cited link. However , I run into this error:

ValueError: Num gradients 1 generated for op name: "mask/Mask"
op: "PyFunc"
input: "conv2/Relu"
input: "Placeholder_2"
input: "Placeholder_3"
attr {
  key: "Tin"
  value {
    list {
      type: DT_FLOAT
      type: DT_FLOAT
      type: DT_FLOAT
    }
  }
}
attr {
  key: "Tout"
  value {
    list {
      type: DT_FLOAT
    }
  }
}
attr {
  key: "_gradient_op_type"
  value {
    s: "PyFuncGrad302636"
  }
}
attr {
  key: "token"
  value {
    s: "pyfunc_0"
  }
}
 do not match num inputs 3

In case needed, this is how I define our_grad function to calculate gradients.

def our_grad(cus_op, grad):
  """Compute gradients of our custom operation.
   Args:
    param cus_op: our custom op tf_mask
    param grad: the previous gradients before the operation
   Returns:
    gradient that can be sent down to next layer in back propagation
    it's an n-tuple, where n is the number of arguments of the operation   
  """
    x = cus_op.inputs[0]
    labels = cus_op.inputs[1]
    epoch_ = cus_op.inputs[2]   
    n_gr1 = tf_d_mask(x)
    n_gr2 = tf_gradient2(x, labels, epoch_)

    return tf.multiply(grad, n_gr1) + n_gr2

And the py_func function (the same as the cited link)

def py_func(func, inp, tout, stateful=True, name=None, grad=None):
    """
    I omitted the introduction to parameters that are not of interest
    :param func: a numpy function
    :param inp: input tensors
    :param grad: a tensorflow function to get the gradients (used in bprop, should be able to receive previous 
                gradients and send gradients down.)

    :return: a tensorflow op with a registered bprop method
    """
    # Need to generate a unique name to avoid duplicates:
    rnd_name = 'PyFuncGrad' + str(np.random.randint(0, 1000000))
    tf.RegisterGradient(rnd_name)(grad)
    g = tf.get_default_graph()
    with g.gradient_override_map({"PyFunc": rnd_name}):
        return tf.py_func(func, inp, tout, stateful=stateful, name=name)

Really need the community's help!

Thanks!

score 3 · Accepted Answer · answered Nov 20 '17 at 01:41

Alright after some efforts I solved this problem myself.

The error message was generated in tensorflow function gradients_impl.py.

def _VerifyGeneratedGradients(grads, op):
  """Verify that gradients are valid in number and type.

  Args:
    grads: List of generated gradients.
    op: Operation for which the gradients where generated.

  Raises:
    ValueError: if sizes of gradients and inputs don't match.
    TypeError: if type of any gradient is not valid for its input.
  """
  if len(grads) != len(op.inputs):
    raise ValueError("Num gradients %d generated for op %s do not match num "
                     "inputs %d" % (len(grads), op.node_def, len(op.inputs)))

Now we can see this error means for our op Mask, we gave it three inputs. TensorFlow expects us to give three output for the function that generates gradients our_grad, each output w.r.t each input for the function for forward propagation.

Note that actually we only want the gradients for the first input, and the other two gradients will not be used anymore in the back propagation, wecan just return two more fake_gradients with the right shape.

ValueError: Num gradients 1 generated for op name: "mask/Mask"

1 Answers1