1

I had asked a question and was implementing the solution when I found out that the operation tf.math.count_nonzero does not have gradient defined. So I tried the following round about method:

eps = 1e-6
a = tf.ones((4, 4, 2, 2), tf.float32)
h = tf.linalg.svd(a, full_matrices=False, compute_uv=False)
cond = tf.less(h, eps)
h = tf.where(cond, tf.zeros(tf.shape(h)), h)
i = tf.reduce_sum(h, axis=-1)
j = h[:, :, 0]
rank_mat = tf.multiply(2., tf.ones((4, 4)))
cond = tf.not_equal(i, j)
rank_mat = tf.where(cond, rank_mat, tf.ones(tf.shape(rank_mat)))
cond = tf.equal(i, tf.zeros(shape=tf.shape(i), dtype=tf.float32))
rank_mat = tf.where(cond, tf.zeros(tf.shape(rank_mat)), rank_mat)
min_rank = tf.reduce_min(rank_mat)

Still the same error persists. I partly understand why this is happening, but is there a differentiable way of implementing this? Thanks.

learner
  • 3,168
  • 3
  • 18
  • 35
  • What exactly are you trying to achieve? You want to increase or decrease the rank of a matrix through optimization? – jdehesa Feb 11 '20 at 13:45
  • @jdehesa increase, the loss function would ultimately be `-min_rank` – learner Feb 11 '20 at 13:46
  • 1
    Maybe you could try to maximize the absolute value of the the smallest singular value, or all the singular values smaller than epsilon, or only the greatest one among those (all compared in absolute value)... not sure if it would work tbh. I'm thinking that may not be very stable, as a near-zero value may dance between positive and negative, but you may try I guess. – jdehesa Feb 11 '20 at 13:50
  • @jdehesa sure, the effect of magnitude of singular values on rank is something that I need to check. But there is no way we can make this differentiable right? – learner Feb 11 '20 at 13:56
  • @jdehesa thanks a lot :) – learner Feb 11 '20 at 14:06
  • 1
    Well, not like you are doing, in the line `rank_mat = tf.where(cond, rank_mat, tf.ones(tf.shape(rank_mat)))` you are setting `rank_mat` to a mixture of two constant matrices, so the gradient is lost. The thing is, you are considering an integer amount (the rank), and differentiation is inherently real (float), I'm not sure what differentiable expression could give you a gradient to "push the rank from 1 to 1.2" (?), so to say. You can use [`tf.custom_gradient`](https://www.tensorflow.org/api_docs/python/tf/custom_gradient) if you have some idea, but I'm not sure what you could do. – jdehesa Feb 11 '20 at 14:07
  • @jdehesa, [this](https://stackoverflow.com/questions/33714832/matrix-determinant-differentiation-in-tensorflow) question does a similar thing for determinant (I agree that the determinant need not be an int). Also, I do realize that the way I have done it is not differentiable. Thanks again. What I'm trying to achieve is that the `4x4` matrix of ranks need all be 2's. – learner Feb 11 '20 at 15:10
  • Hi @learner, can you please include the error log for your case? – TF_Support Apr 07 '20 at 01:18

0 Answers0