14

I am starting to use tensorflow (coming from Caffe), and I am using the loss sparse_softmax_cross_entropy_with_logits. The function accepts labels like 0,1,...C-1 instead of onehot encodings. Now, I want to use a weighting depending on the class label; I know that this could be done maybe with a matrix multiplication if I use softmax_cross_entropy_with_logits (one hot encoding), Is there any way to do the same with sparse_softmax_cross_entropy_with_logits?

Maxim
  • 52,561
  • 27
  • 155
  • 209
Roger Trullo
  • 1,436
  • 2
  • 10
  • 19

3 Answers3

24
import  tensorflow as tf
import numpy as np

np.random.seed(123)
sess = tf.InteractiveSession()

# let's say we have the logits and labels of a batch of size 6 with 5 classes
logits = tf.constant(np.random.randint(0, 10, 30).reshape(6, 5), dtype=tf.float32)
labels = tf.constant(np.random.randint(0, 5, 6), dtype=tf.int32)

# specify some class weightings
class_weights = tf.constant([0.3, 0.1, 0.2, 0.3, 0.1])

# specify the weights for each sample in the batch (without having to compute the onehot label matrix)
weights = tf.gather(class_weights, labels)

# compute the loss
tf.losses.sparse_softmax_cross_entropy(labels, logits, weights).eval()
mauna
  • 1,098
  • 13
  • 25
  • Please, Help! How could I pass `weights` to my_custom_model?https://stackoverflow.com/questions/49312839/how-to-set-parts-of-positive-samples-weight-in-tensorflow-for-binary-classficati – user3151261 Mar 16 '18 at 11:35
2

Specifically for binary classification, there is weighted_cross_entropy_with_logits, that computes weighted softmax cross entropy.

sparse_softmax_cross_entropy_with_logits is tailed for a high-efficient non-weighted operation (see SparseSoftmaxXentWithLogitsOp which uses SparseXentEigenImpl under the hood), so it's not "pluggable".

In multi-class case, your option is either switch to one-hot encoding or use tf.losses.sparse_softmax_cross_entropy loss function in a hacky way, as already suggested, where you will have to pass the weights depending on the labels in a current batch.

Maxim
  • 52,561
  • 27
  • 155
  • 209
1

The class weights are multiplied by the logits, so that still works for sparse_softmax_cross_entropy_with_logits. Refer to this solution for "Loss function for class imbalanced binary classifier in Tensor flow."

As a side note, you can pass weights directly into sparse_softmax_cross_entropy

tf.contrib.losses.sparse_softmax_cross_entropy(logits, labels, weight=1.0, scope=None)

This method is for cross-entropy loss using 

tf.nn.sparse_softmax_cross_entropy_with_logits.

Weight acts as a coefficient for the loss. If a scalar is provided, then the loss is simply scaled by the given value. If weight is a tensor of size [batch_size], then the loss weights apply to each corresponding sample.

Community
  • 1
  • 1
Alexa Greenberg
  • 35
  • 3
  • I was wondering if there was a way to avoid the one hot labels;because in the link provided, there is still need to multiply the matrix of one hot labels with the weight vector. Another way would be using directly the weight vector of length batchsize, but then I would have to compute this vector for every batch; how could I define it (since it depends on the labels) without having to compute the onehot label matrix? – Roger Trullo Oct 24 '16 at 03:37
  • 3
    I don't think this answer is correct. The weights in `tf.contrib.losses.sparse_softmax_cross_entropy` is per-sample, not per-class. – andong777 Apr 15 '17 at 14:13
  • It is correct, it's just annoying. You would pass a weight for each update and that would depend on the particular class that is in the current update. So if you had a batch of size 3 and the classes were 1,1,2. And you wanted to weight class 1 at 50%, then you would use this loss function and pass the weight argument a tensor with values [0.5,0.5,1.0]. That would effectively weight your class... Elegant? No. Effective yes. – Matt S Aug 15 '17 at 17:51