-1

I have a logistic regression model which I created referring this link

The label is a Boolean value (0 or 1 as values).

Do we need to do one_hot encode the label in this case?

The reason for asking : I use the below function for finding the cross_entropy and loss is always coming as zero.

def cross_entropy(y_true, y_pred):
    y_true = tf.one_hot([y_true.numpy()], 2)
    print(y_pred)
    print(y_true)
    loss_row = tf.nn.softmax_cross_entropy_with_logits(labels=y_true, logits=y_pred)
    print('Loss')
    print(loss_row)
    return tf.reduce_mean(loss_row)

EDIT :- The gradient is giving [None,None] as return value (for following code).

def grad(x, y):
with tf.GradientTape() as tape:
    y_pred = logistic_regression(x)
    loss_val = cross_entropy(y, y_pred)       
return tape.gradient(loss_val, [w, b])

Examples values

loss_val => tf.Tensor(307700.47, shape=(), dtype=float32)

w => tf.Variable 'Variable:0' shape=(171, 1) dtype=float32, numpy= array([[ 0.7456649 ], [-0.35111237],[-0.6848465 ],[ 0.22605407]]

b => tf.Variable 'Variable:0' shape=(1,) dtype=float32, numpy=array([1.1982833], dtype=float32)

Relativity
  • 6,690
  • 22
  • 78
  • 128
  • In case of binary logistic regression, you don't required one_hot encoding. It generally used in multinomial logistic regression. – bsquare Apr 19 '20 at 06:17
  • In the reference, fashion_mnist data set has 10 labels (i.e 0-9) hence it required one_hot encoding where as in your case it has only two labels (ex: Yes :1 or No : 0 or vice versa) so it doesn't required one_hot encoding. – bsquare Apr 19 '20 at 06:22

2 Answers2

1

In case of binary logistic regression, you don't required one_hot encoding. It generally used in multinomial logistic regression.

bsquare
  • 943
  • 5
  • 10
1

If you are doing ordinary (binary) logistic regression (with 0/1 labels), then use the loss function tf.nn.sigmoid_cross_entropy_with_logits().

If you are doing multiclass logistic regression (a.k.a softmax regression or multinomial logistic regission), then you have two choices:

  1. Define your labels in 1-hot format (e.g. [1, 0, 0], [0, 1, 0], ...) and use the loss function tf.nn.softmax_cross_entropy_with_logits()

  2. Define your labels as single integers (e.g. 1, 2, 3, ...) and use the loss function tf.nn.sparse_softmax_cross_entropy_with_logits()

For the latter two, you can find more information in this StackOverflow question:

What's the difference between sparse_softmax_cross_entropy_with_logits and softmax_cross_entropy_with_logits?

stackoverflowuser2010
  • 38,621
  • 48
  • 169
  • 217
  • I am getting zero value for tf.nn.softmax_cross_entropy_with_logits(). could you please tell me what can be the issue. – Relativity Apr 20 '20 at 04:10
  • It's possible your data is not in the form that you expect it to be. Print out their shapes, types, and values and make sure they're what the function call expects. Compare shapes, types, and values with the examples in the TensorFlow tutorial. – stackoverflowuser2010 Apr 20 '20 at 05:45
  • thank you. now the loss function is working with tf.nn.sigmoid_cross_entropy_with_logits(), Now the return tape.gradient(loss_val, [w, b]) is returning [None, None] – Relativity Apr 20 '20 at 21:35
  • When we use tf.nn.sigmoid_cross_entropy_with_logits(), the loss will be a tensor with single value in it ( tf.Tensor(636812.56, shape=(), dtype=float32) ). And when we get the gradient of the loss value with respect to weights and b, it is always zero. – Relativity Apr 20 '20 at 23:01
  • 1
    Follow the examples here: https://github.com/nlintz/TensorFlow-Tutorials/blob/master/02_logistic_regression.py In fact, this github repo has a lot of good TensorFlow examples. – stackoverflowuser2010 Apr 20 '20 at 23:08
  • 1
    Also here: https://web.stanford.edu/class/cs20si/2017/lectures/notes_03.pdf – stackoverflowuser2010 Apr 20 '20 at 23:12
  • the examples are using tensor 1.0. (and I am trying in 2.0). I have a question, is there a format for the inputs of gradient function? – Relativity Apr 21 '20 at 00:00
  • What "gradient function" are you referring to? – stackoverflowuser2010 Apr 21 '20 at 00:20
  • If you see the link I have specified. I am referring to the "return" value of "with tf.GradientTape() as tape:" – Relativity Apr 21 '20 at 00:47
  • I'm not familiar with that API. Sorry. – stackoverflowuser2010 Apr 21 '20 at 04:16